Cleaner Way to Decode Missing JSON Keys?

Using Swift
,
1

I have a Codable class that looks like this:

import Foundation

struct MyClass: Decodable {
    let name: String
    let time: Date
    let sometimesPresent1: Double?
    let sometimesPresent2: Double?
    let sometimesPresent3: Double?

    enum DecodingKeys: String, CodingKey {
        case name, time, sometimesPresent1, sometimesPresent2, sometimesPresent3
    }

    public init(from decoder: Decoder) throws {
        self.time = Date()
        let container = try decoder.container(keyedBy: DecodingKeys.self)
        name = try container.decode(String.self, forKey: .name)
        // TODO: Is there any cleaner way to write the below?
        do { // These keys are sometimes altogether missing from the JSON returned by the exchange
            self.sometimesPresent1 = try container.decode(Double?.self, forKey: .sometimesPresent1)
        } catch {
            self.sometimesPresent1 = nil
        }
        do {
            self.sometimesPresent2 = try container.decode(Double?.self, forKey: .sometimesPresent2)
        } catch {
            self.sometimesPresent2 = nil
        }
        do {
            self.sometimesPresent3 = try container.decode(Double?.self, forKey: .sometimesPresent3)
        } catch {
            self.sometimesPresent3 = nil
        }
    }
}

The keys sometimesPresent 1, 2, and 3 are keys that are sometimes provided in the data and sometimes missing altogether. It's also possible the key will be present with the value null.

Is there any cleaner way to deal with keys that may or may not be present in the JSON? My actual use case has 5 such keys so the code becomes very verbose with five do/catch statements stacked on top of one another.

2

This is what decodeIfPresent is designed to handle.

3 Likes
3

Ah, that's it exactly, thank you!

Doc here.

1 Like
4

Can we apply decodeIfPresent for all child properties with a little code , or specially to a property wrapper ?

5

Codable synthesis already uses decodeIfPresent for all Optional child properties. Is there a use-case you have in mind regarding your question here?

6

NO, it doesn't use decodeIfPresent for all optional children , take a look at this use-case

import Foundation

class AES {
    func decrypt(text: String, key: String) throws ->  String {
        return text + key
    }

    func encrypt(text: String, key: String) throws  ->  String {
        return text + key
    }
}

@propertyWrapper
public struct Crypto: Codable {
    private var value: String?
    
    init(value: String?) {
        self.value = value
    }
    
    public var wrappedValue: String? {
        get {
            do {
                return try AES().decrypt(text: value ?? "", key: "DecryptionAESKeys.decryptionAESkey.rawValue")
            } catch {
                return value ?? ""
            }
        }
        set {
            do {
                value =  try AES().encrypt(text: newValue ?? "", key: "DecryptionAESKeys.decryptionAESkey.rawValue")
            } catch {
                value = newValue
            }
        }
    }
    
    public init(from decoder: Decoder) throws {
        if let string = try? decoder.singleValueContainer().decode(String.self) {
            value = string
            print(value)
        } else {
            value = nil
        }
    }
    
    public func encode(to encoder: Encoder) throws {
        if let value = value { try "\(value)".encode(to: encoder)
            
        } else { // encodes to null, or {} if commented
            try Optional<String>.none.encode(to: encoder)
        }
    }
}

struct Item: Codable {
    @Crypto var name: String?
    enum DecodingKeys:  String, CodingKey  {
        case name = "name"
    }
}



let jsonString = """
{}
"""
let data = jsonString.data(using: .utf8)!
let decoder = JSONDecoder()
let item = try! decoder.decode(Item.self, from: data)

it doesn't call the child decode method if the key not found and crash , here I need to it to handle all the Crypto Fields that I have .

7

Ah, thanks for the example. Codable synthesis does use decodeIfPresent for all optional properties, but importantly, annotating a variable with a property wrapper creates a new property with a different underlying type.

When you write

@Crypto var name: String?

the compiler produces the equivalent of

private var _name: Crypto<String?>
var name: String? {
    get { return _name.wrappedValue }
    set { _name.wrappedValue = newValue }
}

In Codable synthesis in the compiler, wrapped properties are treated specially: when the compiler finds the name property, it looks up the backing property for the wrapper:

static std::tuple<VarDecl *, Type, bool>
lookupVarDeclForCodingKeysCase(DeclContext *conformanceDC,
                               EnumElementDecl *elt,
                               NominalTypeDecl *targetDecl) {
  for (auto decl : targetDecl->lookupDirect(
                                   DeclName(elt->getBaseIdentifier()))) {
    if (auto *vd = dyn_cast<VarDecl>(decl)) {
      // If we found a property with an attached wrapper, retrieve the
      // backing property.
      if (auto backingVar = vd->getPropertyWrapperBackingProperty())
        vd = backingVar;

    // ... use `vd` to look up the type of the property, and use that
    // for synthesis.

This means that for the purposes of encoding and decoding, the compiler will encode and decode _name, not name. This is crucial, because it's what allows Crypto to intercept encoding and decoding of the property with its own Codable conformance.

But it's important to note that the type of _name is Crypto<String?>, which is not Optional. You can see this if you try to implement init(from:) on Item the same way that you expect the compiler to:

init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: CodingKeys.self)

    // `name` is a computed property, which can't be assigned to in an initializer.
    // You have to write to `_name`, which is the real property.
    _name = try container.decodeIfPresent(Crypto.self, forKey: .name)
    // ❌ error: Value of optional type 'Crypto?' must be unwrapped to a value of type 'Crypto'
}

If you want to write this, you'll need to support some type of fallback:

struct Item: Codable {
    @Crypto var name: String?
    
    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        _name = try container.decodeIfPresent(Crypto.self, forKey: .name) ?? Crypto(value: nil)
    }
}

This does work the way that you might expect:

let data = """
    {}
""".data(using: .utf8)!
let item = try! JSONDecoder().decode(Item.self, from: data)
print(item) // => Item(_name: main.Crypto(value: nil))

While this behavior is somewhat pedantically correct, the synthesis behavior might not be terribly useful. Theoretically, the compiler could be augmented to do this automatically — it can look through the type of the property wrapper to see if the contained type is Optional, and use a decodeIfPresent then, though this does get complicated in the face of multiple nested wrappers. The biggest risk would be the behavior change.

This might be worth filing as a bug for consideration.

2 Likes
8

Thanks @itaiferber, that explains why the "optional like" wrapper I've built isn't working in the case where the property isn't in the JSON. … the synthesised code for decoding conformance of the containing object doesn't know it's an "optional like" thing.

Perhaps a nice way to surface this in the Codable API would be for it to be "opt in" mechanism? A new protocol can be provided which a type can selectively adopt if it is decodable when the property is missing. Maybe modelled on ExpressibleByNilLiteral:

protocol MissingKeyDecodable {
  init(forMissingCodingKey: ())
}

Then Optional wouldn't iteself have special support, it'd just be using the same mechanism anyone else can adopt:

extension Optional: MissingKeyDecodable where Wrapped: Decodable {
  init(forMissingCodingKey: ()) { self = .none }
}

Would there be a need for the Encodable case as well, so a value can omit itself from being encoded at all if it is in its nil / .none state?

Perhaps there are lots of reasons that none of this wouldn't work at all or would be a bad plan!

Thank you again.

1 Like
9

Yeah, off the top of my head, I think a marker protocol like you're suggesting would work just fine. The compiler could key off of conformance to this protocol to decide whether to use decode(_:forKey:) or decodeIfPresent(_:forKey:).

Yes, we'd need an encode-side protocol as well to indicate whether a value is present or not. I see two main options here:

  1. Introduce a protocol with just a boolean value that indicates "should I be encoded?" — if the answer is true, then the value can be passed as-is to encode(_:forKey:); if false, nothing is called; e.g.

    protocol OptionallyEncodable: Encodable {
    var isPresentForEncoding: Bool { get }
}

  2. Introduce a protocol which returns an explicitly-Optional value to encode instead of self. This value can be passed directly to encodeIfPresent(_:forKey:); e.g.

    protocol OptionallyEncodable: Encodable {
    var valueForEncoding: Self?
}

    (This variant can alternatively have an associatedtype describing the type to use for encoding, but that both strays further away from the point of this protocol [not intended for substituting values], and also introduces problems around inheritance and not being able to override associated types.)

These two approaches have trade-offs:

  • (2) more closely mirrors the behavior we have today by passing off encoding decisions to encodeIfPresent, but introduces a protocol with a Self-constraint (or associated types) which could be slightly annoying to work with
    • Though with SE-0309, maybe this isn't much of a problem at all
  • (1) is simpler and closer to being a small marker protocol, but would be a change in behavior that leads to less flexibility for Encoders (by not calling into them at all if the value is not present)
    • encodeIfPresent is supposed to only encode the given value if it is not nil, but the Encoder could theoretically still choose to do something with a nil value (even if just marking "a value was meant to encode here")

Of the two, I'd probably lean toward (2) for backwards-compatibility, though I haven't thought it through too hard, so there may be some hidden downside I'm not aware of.

In all, the protocols could look like:

protocol OptionallyEncodable: Encodable {
    var valueForEncoding: Self?
}

protocol OptionallyDecodable: Decodable {
    init(forMissingCodingKey: ())
}

Either way, we'd want these protocols in the standard library, which would require going through the Swift Evolution process (pitching, writing a proposal + implementation, etc.).

1 Like