[Pre-Pitch] Codable Customization using PropertyWrappers

Evolution Pitches
1

I recently released v1.0 of CodableWrappers (posted here) if you would like to discuss the library). It's features seem possibly worth writing my first Pitch, so I'd like to gauge the interest level in a more fleshed out pitch and if so, discuss how it would work in Swift (or probably the standard library).

Motivation

Currently the only method of adding custom Encoding/Decoding for Codable Types (without a custom implementation) is by adding options to the Encoder/Decoder. Although this is relatively fleshed out in JSON(En/De)Coder, there are still some major pain points.

  1. Options must be set for every (En/De)coder used
  2. The option Types are separate so e.g. dateEncodingStrategy and dateDecodingStrategy have separate implementations and are both required to handle full serialization.
  3. The options aren't Portable and each (en/de)coder must supply their own options. Not even Swift's own PropertyList(En/De)coder has support for the same options as it's JSON cousin.
  4. It's all or nothing. e.g. If a Type or it's children need to use more than a single dateEncodingStrategy they must manage it themselves.

Proposed solution

Add a set of Property Wrappers that customize the Codable implementation at the property level.

Examples

struct MyDataType: Codable {
    @Base64Coding
    var myData: Data // Now encodes to a Base64 String
}

struct MyDateType: Codable {
    @SecondsSince1970DateCoding
    var myDate: Date // Now encodes to SecondsSince1970
}

Customization that requires additional info can be passed with a Generic Type adopting a protocol with the necessary requirements.

struct MyNonConformingValueProvider: NonConformingDecimalValueProvider {
    static var positiveInfinity: String = "100"
    static var negativeInfinity: String = "-100"
    static var nan: String = "-1"
}

struct MyFloatType: Codable {
    @NonConformingFloatCoding<MyNonConformingValueProvider>
    var myFloat: Float // Now encodes with the MyNonConformingValueProvider values for infinity/NaN
}

Detailed design

The relevant source for the library is here and a more technical writeup here. The current design uses a combination of protocols and generic @propertyWrappers to enable concise Property Wrappers and easy extension by adhering to a single protocol. This is then combined with a set of typealiases to define the specific desired Wrappers

Here's a simplified version of the protocols/wrapper:

public protocol StaticCoder: Codable {
    associatedtype CodingType: Encodable
    /// Mirror of `Encodable`'s `encode(to: Encoder)` but in a static context
    static func encode(value: CodingType, to encoder: Encoder) throws
    /// Mirror of `Decodable`'s `init(from: Decoder)` but in a static context
    static func decode(from decoder: Decoder) throws -> CodingType
}

public protocol StaticCodingWrapper: Codable {
    associatedtype CustomCoder: StaticCoder
}

extension StaticCodingWrapper {
    public init(from decoder: Decoder) throws {
        self.init(wrappedValue: try CustomCoder.decode(from: decoder))
    }

    public func encode(to encoder: Encoder) throws {
        try CustomCoder.encode(value: wrappedValue, to: encoder)
    }
}

@propertyWrapper
public struct CodingUses<CustomCoder: StaticCoder>: StaticCodingWrapper {

    public let wrappedValue: CustomCoder.CodingType
    public init(wrappedValue: CustomCoder.CodingType) {
        self.wrappedValue = wrappedValue
    }
}

So for base64 Data...

public struct Base64DataStaticCoder: StaticCoder {

    public static func decode(from decoder: Decoder) throws -> Data {
        let stringValue = try String(from: decoder)

        guard let value = Data.init(base64Encoded: stringValue) else {
            //throw an error
        }
        return value
    }

    public static func encode(value: Data, to encoder: Encoder) throws {
        try value.base64EncodedString().encode(to: encoder)
    }
}

// Full set of Property Wrappers

typealias Base64Encoding = EncodingUses<Base64DataStaticCoder>
typealias Base64Decoding = DecodingUses<Base64DataStaticCoder>
typealias Base64Coding = CodingUses<Base64DataStaticCoder>

typealias Base64EncodingMutable = EncodingUsesMutable<Base64DataStaticCoder>
typealias Base64DecodingMutable = DecodingUsesMutable<Base64DataStaticCoder>
typealias Base64CodingMutable = CodingUsesMutable<Base64DataStaticCoder>

This results in several advantages over what's currently available:

  • It's Declarative. Serialization details can be constrained to the property definition rather than split between (en/de)coders and documentation
  • Extendable. Adding a custom encoding option is as simple as implementing StaticCoder
  • Single declaration. The customization happens at the property level so it applies to all Encoders and Decoders. 1st party or otherwise
  • Enables multiple (de/en)coding strategies within a single data structure without customizing the Codable implementation

Integration

Although this could replace the current options in JSON(En/De)coder, these approaches can coexist without any issues. If desired the relevant APIs could be deprecated in favor of this approach or if the current API is still desired their implementation could also be re-routed to use the StaticCoders added by the new approach.

Possible Roadblocks

Property Wrapper Ergonomics

Since a Property Wrapper decides the mutability of it's wrapped Property, separate mutable versions of each property are required to support all use cases. e.g. Base64Coding and Base64CodingMutable. I can see this being improved at some point but I haven't seen any indication if/when that would be taking place.

Possible API Confusion

The naming around this is...tricky. I can see potential confusion between the new APIs and the current Codable APIs. If this were to be added to the language some iteration on naming will likely be necessary.

Alternatives Considered

If it's preferable not to expose all the Protocols/Types currently included as public, each Property Wrapper could be a full implementation with the encoding/decoding done directly in the Wrapper (or just StaticCodingWrapper removed). It would be more implementation heavy but could be done via code generation. This would remove the "Extendable" advantage, but would retain the others.

Future Directions

One ability I've been itching for is to customize a Property's CodingKey with Attribute syntax rather than having to override and maintain the entirety of CodingKeys.


struct MyType: Codable {
    @CodingKey("first-name")
    var firstName: String
}

As far as I can tell, this is not currently possible with a Property Wrapper (if someone knows otherwise, please let me know how!). The simplest implementation I can see would be a step in CodingKeys generation that uses the passed name instead of the default.

Anyway, thanks for reading. Any feedback is appreciated!

27 Likes
2

I'll have to reply to this pitch more broadly later because I like the idea and have done similar experimentation (although I cut my own experimentation short when I realized it was not yet possible to compose Property Wrappers in Swift 5.1).

I think you would be able to do this by overloading both decode<T>(_ type: T.Type, forKey key: Self.Key) and decodeIfPresent<T>(_ type: T.Type, forKey key: Self.Key) to grab a string from your property wrapper and use it in a simple CodingKey type that just wrapped that string (or storing said CodingKey, already initialized, in the property wrapper).

3

That was the limit I couldn't get around without getting super hacky. Decodable's init(from decoder: Decoder) means the passed String needs to be available statically. This could be done by passing a Type similar to NonConformingFloatCoding, but IMO at that point it's not really any easier than overriding CodingKeys. Is there another way I haven't thought of?

4

Hmm, yeah, I just did 5 minutes of prototyping and I think the wall I hit was that the approach I was after does not allow one-off overrides. It works if you want to specify all of the coding keys via property wrappers, but that is much less compelling (it just amounts to decentralizing the CodingKeys enum).

5

Maybe another approach to making the arguments of a propertyWrapper available during encoding/decoding would be to modify the code generated by property wrappers and Codable.

And in addition let the propertyWrapper provide a "Builder" type that builds/decodes/encodes the propertyWrapper

Using the example of CodingKey (renamed to CustomCodingKey to avoid confusion with the CodingKey protocol)

@propertyWrapper
struct CustomCodingKey<Value: Codable> {
    var wrappedValue: Value
    var key: String

    struct PropertyBuilder {
        var key: String

        init(_ key: String) {
            self.key = key
        }
        private struct StringCodingKey: CodingKey {
            let stringValue: String
            let intValue: Int? = nil
        }

        func build() -> CustomCodingKey<Value> {
            return CustomCodingKey(wrappedValue: wrappedValue, key: key)
        }

        func decode<Key: CodingKey>(from decoder: Decoder, forKey key: Key) throws -> CustomCodingKey<Value> {
            let container = try decoder.container(keyedBy: StringCodingKey.self)
            let value = try container.decode(Value.self, forKey: StringCodingKey(stringValue: key))

            return CustomCodingKey(wrappedValue: value, key: key)
        }

    }
}

Given this example

struct MyType: Codable {
    @CustomCodingKey("first-name")
    var firstName: String
}

The generated code could be adapted to something like this

struct MyType: Codable {

    /// instead of trying to construct the propertyWrapper itself this would first attempt
    /// to construct the Builder and fallback to the current behaviour if it is not defined
    private static var _firstNameBuilder = CustomCodingKey<String>.PropertyBuilder("first-name")
    private var _firstName = _firstNameBuilder.build()

    var firstName: String {
        get { _firstName.wrappedValue }
        set { _firstName.wrappedValue = newValue }
    }

    init(from decoder: Decoder) throws {
        self._firstName = try _firstNameBuilder.decode(from: decoder, forKey: .firstName)
    }

}
6

I think something like that could work.

On that strain, I wonder if a simpler solution would be some sort of ‘staticPropertyWrapper’ that generates static properties/methods. My guess is it could use some sort of defined Builder pattern like you have, but my experience with that kind of feature is limited so I’m a bit out of my depth as far as implementation.

7

Immutability at the property level should be rare, so I'm not sure it's worth doubling the API surface and making the common case name longer to support it. If it's deemed necessary to have both, I would prefer to make Base64Coding be the mutable version and find a new set of names for the immutable ones.

1 Like
8

FWIW, almost all my Codable Types are composed entirely of let properties. Maybe your experience is different?

4 Likes
9

For me, it's the logical combination of two observations about idiomatic Swift:

  1. Most types should be value types.
  2. Immutable properties of value types are very rarely useful. Making a new instance of the type with a property or properties changed is identical to just mutating properties directly.

You might disagree with one or the other, though.

3 Likes
10

Specifically, for me it's an indication of "This Type is not intended as a workspace". Many of my Codable Types are for API calls and IMO those Types should be strict reflections of the API. Not allowing mutation discourages other uses.

To follow the rabbit trail...more broadly I've been moving towards an increasingly functional and stateless approach. Mutable properties are even considered by some to be an anti-pattern (I'm only partially onboard with that). This is not to say there aren't other perfectly valid approaches. Imperative, OOP, etc. design patters will always have their use cases. But my perception is the general trend in the industry is toward Transforming over Mutating, so that's how I approached the design :slight_smile:

Regardless, the whole issue can be circumvented by allowing the user to decide rather than the author of the Property Wrapper. But that's probably a discussion for a different thread :relaxed:

5 Likes
11

That argument is not convincing to me.

If you are truly using a functional, non-mutating style, then you are almost certainly working with value types.

And if you are working with value types, then you just have to declare the instance itself as a let to get immutability.

Similarly, if you are chaining functional calls, in Swift the return value of a function is immutable anyway.

So if you want immutability when working with data or function calls, you already have it just by virtue of using value types.

Consequently, the natural design is to declare all members of a struct as var. Doing so has no downside, and it has the upside of convenience in those situations where you do need to mutate.

I personally view the use of let properties in a struct as a code smell. Properties should always be var, and in cases where you need to preserve an invariant, then private(set) is sufficient.

4 Likes
12

First, a Purely Function Language doesn't even have mutation. IMO that's overly idealistic, but it's an approach developed with valid reasons. In that context allowing mutation is not only a code smell, it's not even possible! :slight_smile:

Second, yes any Property within a struct could be a var without violating value semantics. But mutability is also about design. Take this User Type:

/// Sent and Returned from http://www.mySite.com/api/v1/user
struct User: Codable {
    var fullName: String
    var firstName: String
    var lastName: String

    convenience init(firstName: String, lastName: String) {
        self.firstName = firstName
        self.lastName = lastName
        self.fullName = "\(firstName) \(lastName)"
    }
}

Leaving the members as var means mutating any property breaks another. This could be handled the OOP way with custom setters, but the "fullName" is going to be a headache and you now have several more Unit Tests you have to write. Add in middleName, suffix`, etc. and your nice concise Type just turned into 100 lines.

Or...you could just make them let and it's good to go.

IME minimizing API surface area in an application is nearly always a good thing. Fewer things to test, fewer things that can go wrong, and less of a chance someone (especially myself) will use it in unintended ways and break things. Again there are other valid ways to approach this, but there are good reasons to make struct members immutable.

Anyway this definitely not what this maybe-a-pitch is about :smile:. If there's a meaningful majority that prefers the Mutable version being the default that's fine by me. It's definitely not a molehill I'll die on.

5 Likes
13

If you intend fullName to be independent of the other two properties, then yes they should all be var. If you intend it instead to be derived from them, then it should use a computed property.

Regardless, human names are difficult, so they would not be my go-to example for, well, anything except their own complexity.

You are making my case for me. When doing functional things, no mutation is possible. Therefore, having member properties declared as var has no downside. They cannot and will not be mutated, so it does not matter how they are declared.

When doing other types of things, there is still no downside to declaring properties as var. If immutability is desired, then declare the instance itself as let.

However, if mutability is desired, then there is a downside to having let properties, because that prevents mutation even though it is desired.

To recap, there is no downside to using var for member properties, regardless of programming paradigm. There is a downside to using let.

14

That doesn't work when it's coming from a 3rd party. (A server)

As long as your property is a var it can be mutated. An instance being let simply means you're choosing not to mutate it and hoping future maintainers don't naively change it. If you're ok being at the mercy of Murphy's Law that's your choice. If the option's available there's someone who will do it. If you're not accounting for that, it's likely something will break.

  1. The Imperative approach is to manage the state directly
  2. The OOP approach is for the Type to manage it's State internally.
  3. The Functional approach is to not have a state.

Always having var uses #1, private(set) is using #2, but somehow #3 is accomplished at the same time? Seems :fish:y to me :confused:

Anyway, seems like your mind's made up so agree to disagree. :slight_smile:

1 Like
15

It seems that you simultaneously have high expectations that let will (in the case of member properties) forever and always preserve immutability even in the face of future maintainers with different plans, while also having low expectations that let will not (in the case of instance declarations) do exactly the same thing.

You can’t have it both ways.

Either let is a strong guarantee of immutability, in which case using it at the instance level is sufficient and properties are free to be var; or else let is a weak guarantee and future maintainers can easily work around it by, for example, creating a new instance with different values and passing that along instead, in which case the properties might as well have been var all along.

Again, I repeat, when using value types, there is no upside to declaring properties as let, there is only downside.

• • •

Or, to use an example from a classic Swift video, there’s no sense in requiring people to build a whole new house just to change the temperature of their oven.

4 Likes
16

Does property wrappers add cpu/memory overhead ?

Basically you just use property wrappers once (parse data from json). But every time you create / access that value, you need to pay the price for having 2 instances.

17

It seems it doesn't add memory overhead, and no CPU overhead for getter and setter. It does add some instructions for modify, but I don't know enough about assembly to say if that's significant.

2 Likes
18

I wonder if the mutability concern could be resolved by having it as an input into the property wrapper init?

@Base64Coding
vs
@Base64Coding(immutable: true)

In any case, I'm with @Nevin on this debate. Mutable would need to be the assumed default.

But as much as I want to like this pitch, I think the one killer feature here is the @CodingKey() wrapper, and If that can't be solved, it's going to be a glaring omission. I can't count the number of times I've been forced to declare an entire CodingKeys enum retyping every property in a type, in order to override the key for a single property :expressionless:

I think the real problem I have with this overall is that it seems like Custom Attributes (annotations) is the right tool here, not actually Property Wrappers. I may be wrong, but seems to me that if custom attributes was part of the language before Codable was created, then they would have played a role in the creation of that feature.

But because the en/decoding synthesis wasn't written against a system of custom attributes, we're trying to shoe-horn in a solution based on Property Wrappers, and that force-fit is the root cause of a lot of the issues you're encountering (ie: mutable vs. immutable, difficulty w/ coding key synthesis, etc).

I'd personally prefer to see a more ambitious pitch that takes this to the deeper level of enhancing the actual codable synthesis to use a system of compile-time annotations built on custom attributes.

2 Likes
19

I think that would require a magic variable/parameter, which I think can be avoided by changing the rules to:

  • (The current rule) A PropertyWrapper may disallow set
  • If the wrapped Property is set let, the set is not generated and wrappedValue.set is set to private.

I would really like to see this, but I think it requires new features being added to the language. As it is I'm hoping this could simply be added to the standard library. It solves real problems I deal with regularly and when the necessary language features are available, @CodingKey can be added.

20

There are major differences between let for an instance vs let for a property. But again, that's not what this thread is about, so agree to disagree. PM me if you want to keep debating :slight_smile:

1 Like