Decoding Generic structures, casting from T? to T when T itself is Optional

generics
codable
optional
#1

Hello,

I'm decoding values of the following type from JSON:

struct DatedBox<T: Decodable>: Decodable {
    var date: Date
    var value: T
}

I wish that DatedBox<String?> would accept all those JSON inputs:

{ "date": ..., "value": "foo" }
{ "date": ..., "value": null }
{ "date": ... }

The two first JSON are decoded perfectly by the synthesized Decodable implementation.

However, { "date": ... } does not. Since the value property is declared as T, the compiler synthesizes container.decode(T.self, forKey: .value), and this requires that the key value is present.

Instead, we need to call container.decodeIfPresent iff the value type is optional.

No problem, let's write our own init(from:decoder) implementation. This gives the following code, which runs correctly in Swift 4.2, and can be pasted in a playground:

here is my question: is the as! cast from T? to T guaranteed to succeed in all future versions of Swift when T happens to be Optional?

import Foundation

/// Decodable helpers
private protocol _OptionalProtocol { }
extension Optional: _OptionalProtocol { }

struct DatedBox<T: Decodable>: Decodable {
    var date: Date
    var value: T
    
    enum CodingKeys: String, CodingKey { case date, value }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        
        // date is mandatory
        date = try container.decode(Date.self, forKey: .date)
        
        if T.self is _OptionalProtocol.Type {
            // "value" is allowed to be missing if and only if our generic
            // type T is optional. In this case, decode nil.
            value = try container.decodeIfPresent(T.self, forKey: .value) as! T // Is this as! guaranteed to work?
        } else {
            value = try container.decode(T.self, forKey: .value)
        }
    }
}

let jsonDecoder = JSONDecoder()
try! jsonDecoder.decode(DatedBox<String?>.self, from: "{\"date\":0,\"value\":\"foo\"}".data(using: .utf8)!)
try! jsonDecoder.decode(DatedBox<String?>.self, from: "{\"date\":0,\"value\":null}".data(using: .utf8)!)
try! jsonDecoder.decode(DatedBox<String?>.self, from: "{\"date\":0}".data(using: .utf8)!)
(Jordan Rose) #2

Hm, so you're relying on the fact that nil gets converted to .some(nil) here? Yeah, that does seem fishy. The "correct" way to do this would be to try to decode the Optional's wrapped type, which is rather tricky to extract from T.

@itaiferber, @Joe_Groff, do we have any standard recommendations here?

(Jordan Rose) #3

I guess one idea is this:

protocol _OptionalProtocol: ExpressibleByNilLiteral {}
extension Optional: _OptionalProtocol {}

if let OptionalT = T.self as? _OptionalProtocol.Type {
  value = try container.decodeIfPresent(T.self, forKey: .value) ??
    (OptionalT.init(nilLiteral: ()) as! T)
} else {
  value = try container.decode(T.self, forKey: .value)
}

It's not pretty but it does express what you want to do: flatten one level of Optional.

#4

That's super smart. I have been trying to add an initializer to _OptionalProtocol as well, but I was unable to avoid Self or the Wrapped associated type (which made _OptionalProtocol a PAT, and prevented the dynamic type test). I'm going to test your solution in a few minutes. Thanks a lot Jordan!

EDIT: Works as smoothly as expected! :+1: And fear is gone :rainbow:

(Itai Ferber) #5

@jrose's answer is a good one — I was going to suggest an alternative approach that circumvents this from a slightly different direction:

if let type = (T.self as? _OptionalProtocol.Type), !container.contains(.value) {
    value = type.init(nilLiteral: ()) as! T
} else {
    value = try container.decode(T.self, forKey: .value)
}

This is essentially what decodeIfPresent already sort of does, but avoids having to finagle the types around the actual decodeIfPresent call. You might be able to shorten up that nil assignment but either way should work.

#6

Thanks @itaiferber, I'm happy that we have two reference implementations for future readers of this thread!

(Oli Pfeffer) #7

@itaiferber is there a similarly fancy example for the inverse (aka Encodable)?

Been playing around w/ it but to my surprise it appears that if-let syntax doesn't catch nil values?

struct Box2<T: Codable>: Codable {
    var value: T

    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: Keys.self)

        if let value = (value as? _OptionalProtocol).value as? T {
            try container.encodeIfPresent(value, forKey: .value)
        }
    }

    enum Keys: String, CodingKey {
        case value
    }
}

let box2: Box2<String?> = Box2(value: nil)
print(String(data: try JSONEncoder().encode(box2), encoding: .utf8)!) // prints {"value":null}, expected {}
(Itai Ferber) #8

You're seeing this because as? T returns a T? — when T is already Optional (e.g. String?), the value produced is a double-Optional (e.g. String??). The if let unwraps that back to a single Optional (String?), and since what you have here is .some(.none), and value itself is assigned .none (AKA nil).

The encodeIfPresent call up-casts that back to T? (AKA String??) and since the value is again .some(.none), the unwrap inside encodeIfPresent succeeds and .none itself is encoded. The whole cast ends up evaluating essentially to a no-op.

One simple way around this is checking whether the Optional actually contains a value itself, à la:

protocol _OptionalProtocol: ExpressibleByNilLiteral {
    var inhabited: Bool { get }
}

extension Optional: _OptionalProtocol {
    var inhabited: Bool { return self != nil }
}

struct Box2<T : Codable> : Codable {
    var value: T
    
    func encode(to encoder: Encoder) throws {
        var container = encoder.container(keyedBy: CodingKeys.self)
        
        let possibleOptional = value as? _OptionalProtocol
        if possibleOptional == nil /* not an Optional<T> */ || possibleOptional!.inhabited /* is not actually nil */ {
            // No need for encodeIfPresent since the value is guaranteed to be non-null.
            try container.encode(value, forKey: .value)
        }
    }
}

let encoder = JSONEncoder()
let data = try JSONEncoder().encode(Box2<String?>(value: nil))
print(String(data: data, encoding: .utf8)!) // => {}

Note that this will not work for nested levels of optionality, e.g.

let encoder = JSONEncoder()
let data = try JSONEncoder().encode(Box2<String????>(value: .some(.none)))
print(String(data: data, encoding: .utf8)!)

still yields {"value":null}, but the recursive unwrapping inside of Optional.inhabited is an exercise left up to the reader. :slight_smile: