Decoding of optionals missing in json

extracting from another thread. i believe this behaviour is broken and needs fixing:

struct S: Decodable {
    var x: Int
    var y: Optional<Int>
}

let data = "{\"x\": 1}".data(using: .utf8)!
let x = try! JSONSerialization.jsonObject(with: data, options: [])

this must fail, as y is not marked as having default value. it currently doesn't fail in any form (including short-hand Int?). the current behaviour doesn't allow you to opt-opt of this "no key ==> nil" behaviour. the fixed version would make it more flexible and allow to differentiate:

"{ x:1}"
    var: y: Int?       ===>  ERROR *** not the current behaviour
    var y: Int? = nil  ===>  y=nil
    var y: Int? = 3    ===>  y= Optional(3)  *** not the current behaviour

"{ x:1, y: null}"
    var: y: Int?       ===>  y=nil
    var y: Int? = nil  ===>  y=nil
    var y: Int? = 3    ===>  y=null

"{ x:1, y: 2}"
    var: y: Int?       ===>  y=Optional(2)
    var y: Int? = nil  ===>  y=Optional(2)
    var y: Int? = 3    ===>  y= Optional(2)

this is a breaking change. cases similar to "var y: Int? = 3" would be rare in practice. cases similar to "var: y: Int?" would cause runtime errors when "y" is missing in json (desired behaviour).

please note that this issue is separate to the infamous "Pre-pitch: remove the implicit initialization of Optional variables". both full Optional and short T? forms are affected.

3 Likes

I don’t think I love the idea that default values of properties would affect the behavior of the synthesized init(from:), personally. I’ve never had a situation where it was dire to differentiate the β€œpresent but null” and β€œnot present” situations, but in such cases it seems like you should be able to write a custom @RequireExplicitNull property wrapper that achieves the desired behavior.

2 Likes

I do not understand the code block that follows. What language is the code written in, and what are you trying to demonstrate with it?

uncompressed version
func foo(_ json: String) {
	let data = json.data(using: .utf8)!
	let x = try! JSONSerialization.jsonObject(with: data, options: [])
	print(x)
}

---------
// case 1, no default value
struct S: Decodable {
	var y: Optional<Int>
}

// case 1.1, no y in json
foo("{}")
expected behaviour: runtime error "y key is absent"
actual behaviour: prints: { y: nil } 🐞🐞🐞🐞🐞🐞🐞🐞

// case 1.2, null y in json
foo("{\"y\": null }")
expected / actual behaviour: prints: { y: nil }

// case 1.3, y=2 in json
foo("{\"y\": 2 }")
expected / actual behaviour: prints: { y: Optional(2) }
---------
// case 2, default value = nil
struct S: Decodable {
	var y: Optional<Int> = nil
}

// case 2.1, no y in json
foo("{}")
expected / actual behaviour: prints: { y: nil }

// case 2.2, null y in json
foo("{\"y\":null }")
expected / actual behaviour: prints: { y: nil }

// case 2.3, y=2 in json
foo("{\"y\":2 }")
expected / actual behaviour: prints: { y: Optional(2) }

---------
// case 3, default value = 3
struct S: Decodable {
	var y: Optional<Int> = 3
}

// case 3.1, no y in json
foo("{}")
expected behaviour: prints: { y: Optional(3) }
actual behaviour: prints: { y: nil } 🐞🐞🐞🐞🐞🐞🐞🐞

// case 3.2, null y in json
foo("{\"y\":null }")
expected / actual behaviour: prints: { y: nil }

// case 3.3, y=2 in json
foo("{\"y\":2 }")
expected / actual behaviour: prints: { y: Optional(2) }

FWIW, this was an explicit design decision made when designing Codable: an Optional property indicates that a specific property need not be present in the encoded data β€” whether it is not present because the key was missing or the value was null doesn't factor into the default behavior.

You can override this behavior if need be by choosing decode(_:forKey:) over decodeIfPresent(_:forKey:) if you need to disambiguate, but the common case rarely cares and a more permissive default helps more code use a synthesized init(from:).

As you note in your examples, too, default values for variables are actually never taken into account when synthesizing init(from:), and this was purposeful too: a default value applied to a variable is not necessarily a reliable indicator of explicit permission to ignore encoded data. For example, many properties may have sensible defaults outside of decoding data, and may developers might prefer to assign default values to those properties (e.g. to benefit from getting synthesized initializers), but those values might be different from what's expected during decoding. null values may be significant, even if the default is not nil.

So code synthesis is conservative and considers the data you're decoding from to be the source of truth, and if you'd like to customize the behavior, it's absolutely possible to do. (Synthesis can't write any code that you couldn't have written yourself.)

In any case, changing behavior here is likely to be massively behavior-breaking. This is something that we'd need to be extremely cautious about, because this would be behavior-breaking in code that developers did not write themselves.

7 Likes
Terms of Service

Privacy Policy

Cookie Policy