Revisit synthesized init(from decoder:) for structs with default property values

Evolution Pitches
1

Currently, adding a default value to a decodable struct will always override the value, even if it appears in the original JSON.

import Foundation

struct User: Decodable {
  let id: Int
  let name: String = ""
}

let decoder = JSONDecoder()
let user = try! decoder.decode(User.self, from: Data("""
{
  "id": 1,
  "name": "Blob"
}
""".utf8))
user.name // ""

This might be surprising to folks on this forum! It was surprising to me even after using Codable a lot.

The behavior is partially documented, and kinda quietly: it's a feature that allow for structs to synthesize init(from decoder:) and omit certain CodingKeys when a default value exists.

A property omitted from CodingKeys needs a default value in order for its containing type to receive automatic conformance to Decodable or Codable.

(From "Choose Properties to Encode and Decode Using Coding Keys.")

For example, the following compiles just fine:

struct User: Decodable {
  let id: Int
  let name: String = ""

  enum CodingKeys: String, CodingKey {
    case id
  }
}

This becomes more confusing, though, when you add an explicit coding key for a property with a default.

struct User: Decodable {
  let id: Int
  let name: String = ""

  enum CodingKeys: String, CodingKey {
    case id
    case name
  }
}

let decoder = JSONDecoder()
let user = try! decoder.decode(User.self, from: Data("""
{
  "id": 1,
  "name": "Blob"
}
""".utf8))
user.name // ""

This is a runtime bug waiting to happen. I think the compiler should be a bit more strict here (rather than have an opinionated, sometimes confusing default). Let's brainstorm some more reasonable expectations around synthesized init(from decoder:) and structs with default property values. I can picture a few solutions to the problem:

  1. Never synthesize init(from decoder:) for structs with default property values. Harsh but no gotchas.
  2. Only synthesize when an explicit set of CodingKeys are declared and the property with a default value is omitted.
  3. Synthesize when an explicit set of CodingKeys are declared, with different behavior depending on whether or not the property is omitted:
    • If it's omitted, always use the default value.
    • If it's present, fall back to the default value if the property is null or missing.
  4. The same as 3., but additionally synthesize when no CodingKeys are declared, changing the behavior to override default property values when JSON is present at that key and non-null.

For 3. and 4. there's some details to work out around optionality, but I imagine the only sensible way to handle the optional property cascade is to:

  1. If the key exists in the JSON and is null, it should be nil.
  2. If the key is omitted in the JSON, it should use the default.

E.g.:

struct User: Decodable {
  let name: String? = "Blob"
}

let decoder = JSONDecoder()
let user1 = try! decoder.decode(User.self, from: Data("""
{
  "name": null
}
""".utf8))
user1.name // nil

let user2 = try! decoder.decode(User.self, from: Data("{}".utf8))
user2.name // Optional("Blob")

Thoughts?

5 Likes
2

Hi Stephen, thanks for putting this together! Some thoughts:

To clarify here, the default value doesn't override what's in the JSON; understanding what's happening here at the base level might put this behavior into more context.

Let's write out the implementation of User ourselves for a sec, because that's all that Codable synthesis does:

struct User : Decodable {
    let id: Int
    let name: String = ""

    private enum CodingKeys : String, CodingKey {
        case id, name
    }

    init(from decoder: Decoder) throws {
        let container = try decoder.container(keyedBy: CodingKeys.self)
        id = try container.decode(Int.self, forKey: .id)
        name = try container.decode(String.self, forKey: .name)
    }
}

If we try to compile this, we'd immediately see the error:

Untitled 3.swift:12:14: error: immutable value 'self.name' may only be initialized once
        name = try container.decode(String.self, forKey: .name)
             ^
Untitled 3.swift:3:9: note: initial value already provided in 'let' declaration
    let name: String = ""
        ^
Untitled 3.swift:3:5: note: change 'let' to 'var' to make it mutable
    let name: String = ""
    ^~~
    var

The property is immutable and already has a value set. We cannot override it no matter how much we want to. Codable synthesis, then, doesn't assign to the property, because it can't.

This is a little bit reversed — the idea is that if you don't want to encode or decode a value from your payload, you have to omit it from your CodingKeys. In order for the property to be initialized on decode, though, it must then have a default value.

Yes — the idea here is that encoding User would encode only its id and not its name.

I would consider the fact that this compiles without warning a bug. If your type is strictly Decodable and you have an immutable property with a default value and that value is included in the CodingKeys enum, I think we should warn about it. The inclusion of name here isn't helpful and hides the actual behavior.

We can and should improve this behavior if it's causing confusion, as long as we can do so in a backwards-compatible way:

This I think would be a bit too harsh. This is valid behavior and it's perfectly valid to rely on it doing the right thing.

Perhaps — however, it's reasonable to expect to want to encode the value if the type is Codable.

We can't get around the fact that the variable is immutable — having more complex behaviors around something that already appears "magic" enough I think could leave us in a worse spot.

I think one thing we could do is the following — for types with immutable properties that have default values, warn if:

  1. No explicit CodingKeys are given (i.e. you're getting the default behavior without necessarily knowing about it)
  2. The value is exclusively Decodable and the value appears explicitly in the CodingKeys (i.e. you're including something which won't get decoded)

I think case (1) is significantly more common, and we could provide a fix-it:

  1. If you expect to decode the value, make it mutable (at least private(set))
  2. Provide an explicit CodingKeys enum (with the value in it if the type is also Encodable, with the value not in it if the type is only Decodable)

The dangerous thing is having this happen to you implicitly without knowing about it; being informed of what's going on would help, I think.

6 Likes
3

@itaiferber Thanks for the thorough and detailed response! (And thanks for all your work on Codable!)

You cleared up a lot of my misconceptions around the behavior here, and I was happy to see that var made defaults work just great. I'm gonna let things simmer a bit and will respond directly soon. I write a lot of Swift with a lot of Codable, but still managed to not understand the behavior I was seeing, so I definitely think there's room for improving the diagnostics here.

1 Like
4

@itaiferber For mutable properties with default values, would it be reasonable for the synthesised init(from:) to allow those properties to be missing from the payload (by decoding with .decodeIfPresent rather than .decode)?

For example, currently this will throw a keyNotFound error, rather than falling back on bar's default value:

import Foundation

struct S : Decodable {
  let foo: Int
  var bar = "default"
}

let jsonData = Data("""
{"foo": 56}
""".utf8)

do {
  let decoded = try JSONDecoder().decode(S.self, from: jsonData)
  print(decoded)
} catch {
  print(error)
}

// keyNotFound(CodingKeys(stringValue: "bar", intValue: nil), Swift.DecodingError.Context(codingPath: [], debugDescription: "No value associated with key CodingKeys(stringValue: \"bar\", intValue: nil) (\"bar\").", underlyingError: nil))

In order to allow us to allow us to fallback on the default, we currently have to implement init(from:) ourselves:

struct S : Decodable {
  let foo: Int
  var bar = "default"

  private enum CodingKeys : CodingKey {
    case foo, bar
  }

  init(from decoder: Decoder) throws {
    let container = try decoder.container(keyedBy: CodingKeys.self)

    self.foo = try container.decode(Int.self, forKey: .foo)

    if let bar = try container.decodeIfPresent(String.self, forKey: .bar) {
      self.bar = bar
    }
  }
}
1 Like
5

This is something that's been brought up before, but the presence of a default value doesn't always indicate that you'd want to use it in case the value in the Codable payload is missing. The semantics depend on why the default value is there — I've seen a lot of code out there which specifies default values solely to avoid having to write out an initializer, e.g.

struct S1 {
    var val: String
    init(val: String = "hello") {
        self.val = val
    }
}

// vs.

struct S2 {
    var val = "hello"
}

let s1 = S1()
let s2 = S2()
print(s1.val == s2.val)

On the other hand, the default value could be integral to the design of the value. Consider also:

struct Person {
    var id = UUID()
    let name: String
}

let jd = Person(name: "John Doe")
// ...

Every time you create a new person yourself, you might want to ensure the person is unique by generating a new UUID. Loading from stored data, though, you want to use the identifier in the payload; if it's missing, you'd silently create a new person here. This is impossible to guess from the structure of the code alone, though.

At the compiler level, we don't know what the intent of providing the default value is — just because there is one doesn't mean it's necessarily intended to be used as the default in case the field is missing.

3 Likes
6

@itaiferber @stephencelis I discussed the issue of immutable properties with default values extensively with @Chris_Lattner3 while working on the flexible memberwise initialization proposal.

There are many good reasons to want the semantics to be such that the default value is only used when an initializer does not explicitly provide a value. The value of such properties would still be immutable but they may have more than one path to initialization and therefore may take on different values in different instances of the type. It sounds to me like this is the model @stephencelis had in mind when posting to this thread. His post is a great example of why we may wish to reconsider the semantics.

My proposal was deferred but the core team indicated willingness to at least consider modifying the semantics of immutable properties. See common feedback and points in the rationale for the core team's decision. If anyone is interested in the discussion between Chris and I, see the draft thread. There was also extensive discussion in the review thread.

The good news is that aside from the initializers synthesized by Codable I don't think there would be any source breakage involved in such a change. We would simply be making code that is not valid today valid in the future. I would really love to see this topic revisited. Perhaps now is the time.

3 Likes
7

True! I don't disagree. I feel strongly about two things here:

  1. We worked hard to make sure Codable isn't magic — it does not synthesize anything you cannot write yourself today. If something is important enough for Codable to synthesize, I think it should be possible for you to write yourself
  2. If and when Swift does give the tools to be able to express this yourself, I'd want to make sure that the rules for the underlying behavior are clear and understandable; this is already a failure spot, so I'd want to make sure we're providing some guidance to push people in the right direction

Perhaps so! I think the discussion of the underlying mechanisms should be revisited if we want to go in this direction; we'd need to make sure there's a clear migration path for Codable types without too much breakage, too.

2 Likes