JSONDecoder, keyDecodingStrategy issue with Dictionary

I am currently converting a code base to Swift 4.1 and I am very happy to be removing CodingKeys implementations everywhere where I previously manually ‘mapped’ keys to snake case.

I was caught a bit by surprise when I tried to decode a Dictionary<String, String> as a value in my model.
The keys in this Dictionary are also decoded from snake case.

A small code example suitable for a playground:

struct MyType: Codable {
    let keyStore: [String: String]
}

let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase

let jsonString = """
{
  "key_store" : {
    "-L5Fo5eYaUsN6OUBfe0": "value",
    "-L5Fo5eYaUsN6OU_Bfe0": "value"
  }
}
"""

let data = jsonString.data(using: .utf8)!
let myType = try! decoder.decode(MyType.self, from: data)
print(myType)

Prints:

MyType(keyStore: ["-l5fo5eyausn6ouBfe0": "value", "-L5Fo5eYaUsN6OUBfe0": "value"])

So my goal is to have the key_store key mapped to keyStore. This works correctly.
But I consider the value of the keyStore variable to be data and would like the contents to be untouched. But as can be seen - when one of the dictionary keys contain an underscore, the key is interpreted as a snake case version of a key.

Looking at the Dictionary conditional conformance to Decodable, it is obvious that this happens, since it just calls decoder.container(keyedBy: ...) public method, and this decoder has it’s keys converted from snake case, so there goes…

So from some perspective, I guess that the decoder is doing what it is supposed to, but it was not at all what I expected. I guess my main point is that I see my model objects with their CodingKeys as part of my model, and I see both the Dictionary keys and values as data and not as part of the model.

Does anyone have any input as to whether this is intentional or an oversight, or if there might be a workaround for this?

@Tony_Parker perhaps? :slight_smile:

1 Like

@Tony_Parker can add to this, but I would say that despite being surprising, this is indeed the correct behavior. Encoding and decoding strategies affect values globally throughout your payload; in the same way that the .dateDecodingStrategy would affect how you might decode a [Date : ...], changing the interpretation of keys affects dictionary keys as well.

From the perspective of an Encoder/Decoder, there’s no distinguishable difference between what you consider your model vs. what you consider your data, short of explicitly overriding specific types to consider them “model” or “data”. It’s possible for us to override the behavior for dictionaries specifically to disobey the strategy, but this would be inconsistent with how strategies are applied today.

In general, the decoding strategies are offered as a convenience for common translations (e.g. if all of your dates are guaranteed to be in one format, it’s easier to apply a .dateDecodingStrategy than to parse those dates manually at every level), but for custom needs, your best bet is to either use .custom strategies to conditionally apply them, or to customize your init(from:)/encode(to:) to suit your needs.

@Tony_Parker, thoughts?

[See also SR-7360]

The ‘strategy’ approach we use tries really hard to be consistent across the entire archive. We felt that if we didn’t do that, it would quickly get confusing about which parts of the payload had which strategies applied to it.

One option is for us to expose the snake/camel conversion functions as static func on the enum, which would make it easier to invoke them in the process of writing a “custom” strategy. That idea did come up during the original review.

I guess that it is not easy to argue against consistency, but I’ll try anyway. :slight_smile:

If we focus on the code and not the archive/payload, it appears to me that there is a very big difference between the CodingKeys of the model objects, and then the keys in a Dictionary that is encoded as a value in the model.
For the model objects you completely control the CodingKeys (you can re-map them in your CodingKeys implementation), while the keys of a Dictionary is something where you explicitly do not have control over the keys.
To me this difference suggests that they are two different concepts and should not necessarily be treated the same. In the first concept you are mapping to and from a model object and in the other you are dealing with data - there is no obvious target to map to.

So even though I follow the consistency argument, I do not see what benefit the user gets from this consistency - at least in any situation where the user considers a Dictionary to represent a value. And when would it not be considered to represent a value?

Apologies for not bringing up these points when the key coding strategies were not initially discussed.

Another way of phrasing my issue with the consistency argument:

One reason for introducing the key coding strategies is to be able to avoid boilerplate CodingKeys implementations throughout the model objects.
If this is the sole purpose of the key coding strategies, then they should only apply to the model objects that implement CodingKeys (synthesized or otherwise), right?

Forgive me if there are other use cases or complications that I am not seeing. :slight_smile:

With regards to implementing custom strategies, they would need to be based on knowledge about the path of keys from the top level object and to the current value to decide on the strategy. This means that if one were to skip the snake case strategy from all Dictionaries, one would need to know all paths to Dictionaries present in the all model objects. This seems a bit backwards.

I would rather try experimenting with a way of opting out from the key coding strategies by conforming types to a protocol.

4 Likes

I’m not sure about this, but maybe allowing modification of the decoder options during decoding (inside the init(from:) method) would solve this. Allowing this for all options would also address similar issues that might come up for the other options of JSONDecoder.

Example

The custom decoding code would look somehow like this:

init(from decoder: Decoder) throws {
    // decode things with the default key decoding strategy 
    // set key decoding strategy to the desired one, storing the old one
    // decoding in this example the dictionary
    // set strategy back to the old strategy
}

Implementation

  1. This could be implemented by either exposing _JSONDecoder and JSONDecoder._Options publicly and making options on _JSONDecoder a var.
  2. or by (hypothetically) storing the options inside userInfo (currently, it is the other way around) at a key like JSONDecoderOptions that would be defined by JSONDecoder and contained the _Options struct as value. However, this is purely theoretical, because Decoder has userInfo declared only as as get property. This approach also seems to turn around the sense of userInfo a bit, because here it’s also information from the user to the decoder and not information purely for the user. However, I see this way as the most elegant one, but requiring either source breaking changes or additional API, like a options: [OptionKey : Any] { get set } property on Decoder with default implementations returning [:] and doing nothing.
  3. Forward changes on the options (would need to be public, other option properties would be removed) of JSONDecoder to _JSONDecoder. Every _JSONDecoder would store a reference to the JSONDecoder that created it and read those options or _Options would explicitly use reference semantics, such that _JSONDecoder and JSONDecoder could use the same _Options reference, both having own options properties. Making _Options would do this, I guess. In general, in this case there could be a bit additional confusion why I would need to store the JSONDecoder and access it in my init(from:) method somehow, if I get another decoder as argument anyway.

Alternative

This is a bit tricky alternative, that is already implementable today.
If one already made sure, that one is decoding from JSON, e.g. by checking a previously set key in userInfo, one decodes a JSONValue enum to get unevaluated information. Then one will convert this JSON types consumable by JSONSerialization, re-encode the unevaluated information and then pass this newly created JSON data to a new JSONDecoder with the desired options (concretely, a different key decoding strategy) and decode the type (dictionary here) with this JSONDecoder.

This might get a but easier with decoders providing unevaluated information directly, as discussed in New "Unevaluated" type for Decoder to allow later re-encoding of data with unknown structure, but allowing this has not been discussed now, as far as I know.

Because the re-encoding keys does not result in the same keys, as far as I can see, it is necessary to go over JSONSerialization here, I guess. One can use JSONEncoder to, if the used JSON implementation is Encodable, but this is actually one stage more.

Thank you very much for your suggestions.
I did try looking into a solution where it is not necessary to ‘remember’ to have to perform the workaround for all Dictionaries in the model.

My solution is to add an empty protocol (CodingKeyCodingStrategyOptOut) and have Dictionary conform to this.
In the *Storage implementations in JSONEncoder and JSONDecoder I keep a stack of useKeyCodingStrategy that grows and diminishes in parallel with the containers (well, almost).
In ‘box’ and ‘unbox’ of the encoder and decoder respectively, I check if the converted type conforms to my protocol - if so, I push a ‘false’ value to my stack, otherwise ‘true’.
In the coding strategy implementation I then check this value.

This means that I can now make any type opt out of the KeyCodingStrategy. I am only using it for Dictionary, however.

It’s not beautiful, but it works for our use case.

It works correctly with nesting, and with root objects opting out.
Since this method determines the behavior from the type of the encoded/decoded values, it follows naturally that superEncoder and nestedEncoder should inherit the useKeyCodingStrategy state. Since if you are using a nestedEncoder, you are still encoding the same type, etc.

Would anyone consider such an API to be nice to have in Foundation?

This sound a lot more pratical zo me than my suggestion.

Something similar could also be implemented, if a type instead of a coding path would be passed to cusom key decoding strategies, at least in theory, if the key decoding strategy was applied at a later point, in decode and nestedContainer and so on.

However, since as far as I know it isn‘t possible to create a „generic closure“, the closure you pass if using the new customWithType key decoding strategy case would look like (type: Any.Type, codingPath: [CodingKey]) -> CodingKey).
I don‘t know if type checking in the form type is MyMarkerProtocol.Type works if Any.Type is the type of the argument.
Update: It works:

protocol MyMarkerProtocol {}
let closure: (Any.Type) -> Bool = { type in return type is MyMarkerProtocol }

extension Dictionary: MyMarkerProtocol {}
closure(String.self) // returns false
closure([String:Int].self) // returns true

Right, that would be an even better place to handle inclusion or exclusion from having the keys decoded.

I have been experiencing some other issues with keyDecodingStrategies - issues that I think could easily lead to bugs.

These issues have made me think of an entirely different way of performing key mappings.

I think this could perhaps be made into a pitch - and without having actually tried out the implementation yet (it requires features of Swift 4.2), I think that it is doable.

Ok, so here goes my pitch for a pitch for a proposal:

More robust key coding strategies for Encoders and Decoders.

In Swift 4.1, JSONDecoder and JSONEncoder added the options of performing a mapping of keys through the keyDecodingStrategy and keyEncodingStrategy properties.

I see two issues with the current solution.

The first issue is that I found it unclear which key is referred to in the keyEncodingStrategy and keyDecodingStrategy respectively.
One refers to a CodingKey (the keyEncodingStrategy maps from a CodingKey value to a String key) and the other refers to a key in the JSON structure (as the keyDecodingStrategy maps from a String key to a CodingKey value).

This brings us to the second issue. Namely that the above means that we have two mapping functions:
(CodingKey) -> String
and
(String) -> CodingKey

In practise this can lead to unexpected mappings.

Consider the following type:

struct MyType: Codable {
   let imageURL: URL
}

Using a keyEncodingStrategy of .convertToSnakeCase, this will lead to the key:
image_url.

All fine and dandy.

But applying the keyDecodingStrategy of .convertFromSnakeCase will make the conversion fail, since the generated CodingKeys enum will be initialized with the converted string key: "imageUrl", which of course does not exist.

This can of course be remedied by adding a CodingKeys enum as follows:

struct MyType: Codable {
   let imageURL: URL
    enum CodingKeys: String, CodingKey {
        case imageURL = "imageUrl"
    }
}

Now the key decoding matches again, but there is a serious cognitive overhead required here.
Basically the thought process in creating the CodingKey case requires one to perform the conversion to snake case and then convert that back to camel case.

It is not easy.

Further similar examples that I think are harder to grasp than they should be:

Actually trying to map a property name to a different key name:
I want to map the property: internalName to the key: “my_more_elaborate_external_name”
This has to be done like so:

struct MyType: Codable {
    let internalName: String
    enum CodingKeys: String, CodingKey {
        case internalName = "myMoreElaborateExternalName"
    }
}

A final example deals with the situation where you have already manually mapped to snake case. Since the key that will be passed to the CodingKeys initializer when decoding is already converted to camelCase, this will naturally fail.

struct MyType: Codable {
    let alreadyMapped: String
    enum CodingKeys: String, CodingKey {
        case alreadyMapped = "already_mapped"
    }
}

Finally there is the case of model values of type Dictionary. Since a keyDecodingStrategy of .convertFromSnakeCase converts all JSON keys, it currently also converts keys of the Dictionary to camel case. But only if the JSON keys contain underscores. This can cause confusion (bugs) since the data being parsed may not always contain underscores, so you may only experience this a long while after putting code into production.

Consider:

struct Profile: Codable {
    let name: String
    let identifier: String
    let accounts: [String: Bool]
}

Let’s imagine that accounts is intended to hold a map from an auto generated account identifier to a Bool.
Parsing the following JSON structure (based on a true story)

{
  "name": "Morten",
  "identifier": "63bci9n2",
  "accounts": { 
    "-L3SATbsBxyXgju9Pqem": true
    "-L3S5hBZ7elsJ1yX6mQg": true
    "-Kr0epMgHcLs9_koxE1Q": true
  }
}

At the time of actually using an account identifiers, the two first are decoded without any conversion, but the last identifier is decoded to "-kr0epmghcls9Koxe1Q" which may have caused a bit of confusion for the user.

Proposed solution

I propose that there should only be one key mapping function. Namely the one from
(CodingKey) -> String.

When decoding, one should not perform an inverse conversion (which is not always possible, as can be seen in the examples where the source is "imageURL" or "already_snake_cased").
Rather, when the CodingKey-conforming type is iterable, all keys should be converted, and the key used for decoding should be looked up in a map from the converted values to the original.

This has a few implications:

  1. Only types where the keys can be enumerated can have key coding strategies applied. (This will basically fix my issue with Dictionary keys being converted).
  2. The CodingKey can be written as you like. "image_url", "imageURL" and "imageUrl" all map to "image_url" using ‘to snake case’ conversion.

This strategy will solve all the issues described above issues.

(Somewhat) detailed design:

Key encoding and decoding strategies should only be applied to CodingKey-conforming types that are also CaseIterable.

The synthesized CodingKeys structs should be made CaseIterable.

Compiler diagnostics should suggest adding CaseIterable to enums that conform to CodingKey.

Encoding of keys should be performed as today.

Decoding of keys should iterate over all enum values and create a Dictionary<String, T> (where T is the CodingKey-conforming type) map for performing lookups.

Additional considerations

The CodingKey protocol could be given extra methods:

func allowKeyCoding(for encoder: Encoder) -> Bool
func allowKeyCoding(for decoder: Decoder) -> Bool

that could have a default implementation returning true

This would enable users to opt out of key coding for any encoder and decoder - or for specific encoders and decoders.


How does that sound? I would like to give the implementation a shot.
Does anyone think that my pitch pitch should be made into a pitch? :slight_smile:

Thanks for putting so much thought into this, and apologies that I haven’t gotten the chance to respond to comments earlier upthread. There’s a lot to unpack here, so let me address just a small part of this before this gets too far away from me. Two points:

  1. Key mappings are provided as a feature on JSONEncoder/JSONDecoder specifically at the moment as a pragmatic feature to make it easier to conform to some JSON conventions without having to do too much work. The strategies offered on there are meant to be useful for the format, but might not make as much sense on other encoders and decoders; I wouldn’t want to overgeneralize the solution to cover problems that other formats don’t have
  2. I think as far as the original issue here goes (applying these strategies to dictionaries), I think we’ve come around to agree that this is really a bug which we can (and should) fix. Luckily, the fix should be relatively simple — in JSON boxing and unboxing, dictionary behavior can be overridden directly where string keys are not translated into CodingKeys but are written out directly

For now, I think the best solution is to keep this focused to the original problem. If someone is interested in putting up a PR which fixes this behavior for dictionaries, we’d be happy to review it and merge for Swift 4.2.

My pleasure - it’s fun to be a part of the discussion. ;-)

With regards to your two points:

  1. I do think that the robustness of only having a mapping in one direction - instead of two - has a great worth (I think that it is less error-prone). It could be interesting to see if that could be solved in some other way than requiring the CodingKey to be CaseIterable (and thus not having to overgeneralize) at some point in the future.

  2. Excellent. I am really glad that you agree on this being a bug.

I would love to give the implementation of a bug fix a shot.

I did try looking at that option when implementing my workaround, but I’m afraid that I need a bit of help:
How would you constrain this specific unboxing to a Dictionary where you have generic types for keys and values?

If you could point me in the right direction I would be happy to give it a shot!

I don’t disagree — I think that were we to start generalizing key conversion strategies to more encoders, we can explore further down this path.

Happily! Here are the relevant details:

  1. The implementations of Dictionary.encode(to:) and Dictionary.init(from:) are very straightforward. As you can see, CodingKey conversions happen only if the dictionary keys are already String or Int, which makes the application of this strategy easier. If the keys are not Int or String, the dictionary encodes as an array of key-value pairs, to which no key strategy is applied
  2. _JSONEncoder.box_ is the method which takes an existing generic object and boxes it up into the NS representation
  3. Similarly, _JSONDecoder.unbox converts things the other way around

The general idea here would be to add another case in box_ and unbox to look for dictionaries whose keys are Int or String and instead of allowing to encode themselves (and thereby converting the keys to CodingKeys), boxing them up by iterating over the values then and there.

Happy to clarify further if I can.

Thanks - I think it’s the 'look for dictionaries whose keys are Int or String’ that I have trouble implementing.

For instance in unbox - here my Swift-foo is failing me - how would you do the actual test to see if the type is a Dictionary when it has generic parts?

Ah, I forgot. While some Swift types are special and support covariance in their generics at runtime (Optional, Array, and Dictionary special in this regard; not sure about others off the top of my head), we don’t support them at compile time for any type, I believe. So the following works for checking whether a value is a String-keyed dictionary at runtime:

func isStringKeyedDict<T>(_ value: T) -> Bool {
    return value is [String : Any]
}

isStringKeyedDict([1, 2, 3]) // => false
isStringKeyedDict([1: "one", 2: "two"]) // => false
isStringKeyedDict(["hi": 5]) // => true

But you can’t extend this statically for when you don’t have an actual value, but a static type:

func isStringKeyedDict<T>(_ type: T.Type) -> Bool {
    return type is [String : Any].Type
}

isStringKeyedDict([Int].self) // => false
isStringKeyedDict([Int : String].self) // => false
isStringKeyedDict([String : Int].self) // => false

@jrose Is there a nicer solution to this problem these days, or is the cleanest thing at the moment using a marker protocol? The following works, but it isn’t especially pretty:

protocol _JSONStringDictionaryMarker {
    static var isStringKeyed: Bool { get }
}

extension Dictionary : _JSONStringDictionaryMarker {
    static var isStringKeyed: Bool { return Key.self == String.self }
}

func isStringKeyedDict<T>(_ type: T.Type) -> Bool {
    guard let marker = type as? _JSONStringDictionaryMarker.Type else { return false }
    return marker.isStringKeyed
}

isStringKeyedDict([Int].self) // => false
isStringKeyedDict([Int : String].self) // => false
isStringKeyedDict([String : Int].self) // => true

This can be simplified once conditional conformances can be dynamically queried; we could make the marker protocol empty and only extend Dictionary where Key == String:

protocol _JSONStringDictionaryMarker {}
extension Dictionary : _JSONStringDictionaryMarker where Key == String {}

func isStringKeyedDict<T>(_ type: T.Type) -> Bool {
    return type is _JSONStringDictionaryMarker.Type
}

Cute. No, I don’t think we have a better answer yet if you’re starting with a completely unknown type. It would be nice™ to have a language feature for casting to Dictionary<let keyType, let valueType> or something like that, but that needs a lot of design.

2 Likes

Yeah, that would indeed be nice! In this case, even simpler covariance checks would suffice (type is [String : Any].self), but a generalized solution would solve a lot of other problems.

@Morten_Bek_Ditlevsen Seems like a simple (private) marker protocol + conformance should be enough to identify dictionaries like this, then

1 Like

I just had a quick look at a fix. For decoding it appears that a workaround is easy, but for encoding you still need to call encode(_:forKey:) on all key value pairs, and the key conversion happens here.
Could I try fixing it by applying explicit knowledge about the _DictionaryCodingKey type in the key conversion?

Hmm, that approach didn’t work either, as _DictionaryCodingKey is not visible from Foundation.

I don’t have any really ideas except for storing extra information on the _JSONKeyedDecodingContainer and ditto for encoding. This is how I made my own shot at implementing a protocol allowing you to opt out of the key coding strategy.

There is of course a really ugly hack: :slight_smile:

        let useKeyCodingStrategy: Bool = "\(K.self)" != "_DictionaryCodingKey"

Any other suggestions? I guess there’s no way _DictionaryCodingKey could become visible to Foundation?

Ah, sorry, I guess that I can just encode the values directly and not into a keyed container.

Ok, now I think I know what to do. Apologies for my confusion! :slight_smile:

Do I get this right, that we are talking about not applying the key decoding strategy for dictionaries here?

I have a few concerns:

  1. pretty unpredictable outcomes, actually even more than right now in my opinion
  2. no support for custom keyed collection types, though they can of course convert themselves to Dictionaries and encode them
  3. Dictionary is loosing introspection capabilities, e.g. all JSON workarounds encoding or decoding [String : JSON] will no longer work as expected (this is actually 1.)
  4. The key conversion is still uncontrollable. I can not opt in to dictionary key conversion the same as I can not opt out right now.

However, if this decision has already been made, I don’t want to question it right now.

Hmm, I guess that one (wo)man’s bug is another (wo)man’s feature.

It is true that you will not be able to control key conversion for Dictionaries. But you would be able to perform this mapping in init(from decoder:) and encode(to encoder:) manually, whereas with the current behavior, you cannot ‘undo’ the mapping that has already been performed.

With regards to 1. and 3. I would actually think that it is easier to introspect your [String : JSON], since the keys decoded from your JSON are left untouched.