[Pre-pitch] Roundtripping key coding strategies

Hi all,

The current state of key coding strategies with Codable

Today the Codable system has a leaky abstraction if used with encoders and decoders that perform transformations on their keys.

In Foundation this is currently only present in the JSONEncoder and JSONDecoder, but the issue that I'm explaining would be the same for any other encoders/decoders that would attempt to do something similar to JSON(En/De)coders key(En/De)codingStrategy.

The issue is, that there is currently a pair of transformations present - and since the transformations are lossy you don't necessarily get to the source key by encoding and decoding it.

For instance if I set the keyEncodingStrategy of a JSONEncoder to .convertToSnakeCase and a similar keyDecodingStrategy to .convertFromSnakeCase and use it with the following struct:

struct Person: Codable {
  var imageURL: URL
}

Then the encoding transform will produce:

{
  "image_url": "..."
}

But the decoding transform will go from snake case to camel case, trying to look up the key: imageUrl, which does not exist.

This is a common source of bugs when using key coding strategies, and at least in code bases that I am familiar with, the workaround is often to add custom coding keys like:

enum CodingKeys: String, CodingKey {
  case imageURL = "imageUrl"
}

This allows the imageURL property to roundtrip when used with the snake case encoding and decoding, but this is a 'leaky abstraction'.

Codable entities and the encoder/decoder they are used with are supposed to be decoupled, but in this situation, the developer needs to know if the codable entity is used with an encoder/decoder pair that use key transformations - and also need to remember to map the key correctly, so that it will be 'found' when converting back from snake case to camel case.

Often I have seen attempts to 'fix' the behavior with the notation you would use if you didn't apply a key coding strategy:

enum CodingKeys: String, CodingKey {
  case imageURL = "image_url"
}

which of course is no good when used with snake case conversion, since the key that will be searched for is "imageUrl"

Other times I have seen developers thinking that the custom CodingKey implementation must be a mistake and removing it entirely, because unless you are very familiar with both the use case and the peculiarity of this mapping, then the code does look a bit 'off'.

Finally having this custom coding key also means that you are in trouble if you wish to encode/decode the same entity with an encoder/decoder where you are not using a similar key transform.

The road to a solution

So the basic issue is that we can't make a perfect inverse of a lossy transformation.

One solution (credit goes to @norio_nomura who made a PR with a proposal for a solution, that was unfortunately got closed during a PR cleanup: Add useSnakeCasedKeys to JSONDecoder.KeyDecodingStrategy by norio-nomura · Pull Request #14039 · apple/swift · GitHub) is:

  1. Recognize that if you have two lossy functions, a and b, then b(a(input)) won't equal input for all values of input.
  2. If you only have one function and treat the coding keys as the only source of truth, then you can get a perfect mapping.

Fortunately most API that deal with coding keys in the entire Codable surface area have the original coding key as input. This is the brilliance of the suggested PR above: the transformation of the coding keys during both encoding and decoding is actually a transformation on the coding key.

UNfortunately, there's one API where this is not true - namely:

var allKeys: [KeyedDecodingContainer<K>.Key] { get }

on KeyedDecodingContainer.

Since this method only has the "encoded" coding keys as a source, it basically depends on the inverse transformation that the proposed solution tries to get rid of.

Can this issue be solved?

I don't know.

One idea would be to deprecate the allKeys API and introduce a similar allRawKeys method that returns an Array of anonymous CodingKey directly from the input.

For myself - in all the time I've been using Codable, I have not directly been using the allKeys API.

If I have a case of 'dynamically keyed data' I tend to always decode a Dictionary<String, Something>, since decoding a Dictionary implicitly opts out of key decoding (the Dictionary is treated as data, not as a container with coding keys as keys).

One use case I can see for this API is to test if the decoding container only contains the keys that I am interested in, and no others. But that could also be solved by counting the allRawKeys array.

My gut feeling is: If you use key coding strategies together with the allKeys API, are you not already in trouble? In what sort of dynamic situation would it be ok and even expected for you to get back an array of transformed keys using this API?

So this leads me to ask some questions in this forum:

  • How have you used the allKeys API on KeyedDecodingContainer?

  • Have you used it together with key coding strategies?

  • Have you used it for anything other than getting the count of keys?

  • Would you be able to replace your use of allKeys with something like the proposed allRawKeys API? Or even by decoding a Dictionary and dealing with that afterwards?

Final thoughts

A solution to this issue does not only benefit JSONEncoder/JSONDecoder, but any encoder/decoder attempting to map keys in the process.

With a solution in place, you would be able to use almost any function as a transform - for instance your keyCodingStrategy could hash the keys in some way, and still be able to decode the hashed keys into the original coding key. That sounds like a powerful abstraction to me. :-)

Note: I have deliberately not mentioned keys that would clash using a specific key coding strategy:

struct Person: Codable {
  var imageURL: URL
  var imageUrl: URL
}

This situation is neither improved nor made worse by only having a key transform in one direction.

7 Likes

I don't have much to add here beyond what's been discussed already, but some assorted thoughts/links:

  • Lots of good background in the original PR and in the followup thread, which I think are worth reading
  • Gut feeling: this is a problem mostly created by JSONEncoder for itself — I'm not sure how many other encoders allow key modification in this way. It may be entirely sufficient to confine the solution there
    • One thing to keep in mind is that the key encoding strategies currently allow for a .custom strategy, which may not follow the same rules as the strategies one might have in mind. I don't know how often .custom is used, but it is there
  • I think allRawKeys could work (and we may not need to deprecate allKeys if we can just offer the raw version), though adding it at this point would require giving it a default implementation to prevent source breakage
    • You could theoretically just have the property default to returning allKeys, though it may be a trap to forget to implement it

I'd love to be able to solve some of the inconsistency here too — we may need some input from some Foundation folks if they have time to spend on the problem, especially if the solution is constrained to Foundation-specific code (e.g. altering the JSONEncoder/JSONDecoder strategies) instead of changing all containers. (It does appear that the new AttributedString decodable implementation uses allKeys for decoding containers of attributes, so they may have some thoughts about how changes here might affect the implementation.)

1 Like

Hi Itai,
Thank you for your comments - and for adding a link to the followup thread.

I am going to experiment with AttributedString to see if it may already have issues with encoding/decoding properties when using key coding strategies. It may at least have issues if you define custom attributes and use multiple uppercase letters in a row.

I think that in time, the very best outcome would be to deprecate all existing key coding strategies and replacing them with ones that only go in one direction. The .custom one too could be replaced with one that always goes in the direction from key to 'encoded key' on both the encoder and decoder.

I think that even though the issue arises through the implementation in JSONEncoder and JSONDecoder, key coding strategies, the issue is not actually with them, but with the fact that the 'allKeys' API exists (and I know that it exists for a perfectly good reason) - because it's existence basically requires anyone that wants key conversion to implement them as two separate functions instead of one.

1 Like

A quick followup about the AttributedString Codable conformance:

It already doesn't play well with snake case key coding strategy:

let input = AttributedString("Hello").settingAttributes(.init([.imageURL : URL(string: "https://example.com/")!]))
print("Before: ", input)

let encoder = JSONEncoder()
encoder.keyEncodingStrategy = .convertToSnakeCase
let decoder = JSONDecoder()
decoder.keyDecodingStrategy = .convertFromSnakeCase

let encoded = try! encoder.encode(input)
print("Encoded: ", String(data: encoded, encoding: .utf8)!)

let decoded = try! decoder.decode(AttributedString.self, from: encoded)
print("After: ", decoded)

Outputs the following:

Before:  Hello {
	NSImageURL = https://example.com/
}
Encoded:  ["Hello",{"n_s_image_url":{"relative":"https:\/\/example.com\/"}}]
After:  Hello {
}

So it suffers from a similar (but not quite the same) issue as described above, but doesn't fail parsing - it just has an 'empty' attribute covering the "Hello" String.

I think that there may be some learning from this - I hope that I can formulate it clearly:

First of all my intuition about 'dynamic' keys still holds with this example: namely that it doesn't play well with key coding strategies.

Secondly: Is there a way to fix this? The only thing I can think of is a mechanism to explicitly opt-out from key coding strategies. You actually already get that when decoding dictionaries keyed by Strings - the keys are left alone. I suspect that dictionary decoding wouldn't work in this case, however, but perhaps we can think of another way to opt out of the key coding? Another 'marker' protocol on CodingKey, perhaps?

Just wanted to chime in that XMLCoder maintained by @Max_Desiatov also has the concept of key coding strategies.
The Firebase open source iOS sdk uses a JSONEncoder-inspired encoder/decoder pair, that also supports key coding.

Just to say that the general concept is definitely useful across more encoders and decoders, and I believe that a solution to the issue would be welcome in other projects too.

2 Likes

I’m having a bit of an issue figuring out how to focus this pitch:

  1. Just suggest the addition of new key coding strategies that basically implements the ideas presented in the PR linked above.
  2. Suggest an allRawKeys api
  3. Suggest a way to opt-out of key coding (this makes the general concept of key coding into something that the Codable api now needs to know about.

Since applying key coding strategies already breaks round tripping today - for types like AttributedString, then one could argue that 1 could stand alone and improves one area (fixing the leaky abstraction) without incurring additional issues (that are at least conceptually present and may be quite well hidden).

The only thing is: this has already been attempted (the linked PR), but that got stranded in internal api review.

I could go for proposing 2, but I am unsure how much it would actually solve. I don’t think using that api could fix the AttributedString round tripping issue.

Then there is 3. That could potentially be used to fix AttributedString round tripping. The cost is another marker protocol - together with a new thing to learn when implementing (quite advanced) Codable types.

What do you think?

GRDB has it as well. The implementation blindly follows Foundation (because when in doubt I grab inspiration from standards), and is also unable to roundtrip is some specific cases: fooID -> foo_id -> fooId.

1 Like

2 & 3 may work together. Instead of (or in addition to) allRawKeys, you can have allKeys which is available only when key confirms to PreformattedCodingKey.

1 Like

That's a really neat idea!
I'm not certain how this would look at protocol level though. You can extend the functionality of a concrete type based on the protocol conformance of generic parameters, but can you similarly extend the protocol requirement based on the protocol conformance of an associated type?

And also: could this be implemented in a backwards compatible way?

Hi all,
Thank you for the comments so far! :slight_smile:

I think that I am soon ready to formalize the presented ideas.
So basically there are three parts, that can be though of as independent:

  1. Introduce key coding strategies that only go in the direction of the encoding (from CodingKey of the type in question to encoded representation). Deprecate old key encoding and decoding strategies (in Foundation!)
  2. Introduce a new protocol (I like the suggested name: PreformattedCodingKey). Skip key coding strategies for keys conforming to this protocol during both encoding and decoding.
  3. Introduce a protocol requirement to KeyedDecodingContainerProtocol for an allRawKeys: [CodingKey] property. Add a default conformance that returns allKeys for backwards compatibility.

Regarding 1:

My suggestion would be for adding a new enum KeyCodingStrategy that is shared bewteen JSONEncoder and JSONDecoder. The sharing would help making it evident that there is only one transform in play.

Add a new option for JSONEncoder and JSONDecoder called keyCodingStrategy while deprecating keyEncodingStrategy and keyDecodingStrategy for the two types respectively.

The rationale is that even though individual cases can be deprecated, it should be made clear that a new .custom case for the decoding is also in the direction of encoding. Deprecating the existing .custom case would require a new name to be invented. And the argument for sharing strategies between encoder and decoder also suggests that deprecation of the existing strategies would be sensible.

A rather big issue: This is all in Foundation, so basically outside of the realm of SE...

Does anyone know of a good way to aproach this? The previous PR which almost does what I am suggesting here got stranded for two and a half years and then closed.

Note that this step may of course be taken by any non-Foundation encoder/decoder pairs today already.

Regarding 2 and 3:

Both of these could definitely be discussed in SE and respecting the protocol would be natural for both JSONEncoder/JSONDecoder and other encoder/decoder pairs .

It would be natural for types like AttributedString to opt-in to PreformattedCodingKeyfor their codingkeys, but again this lives in Foundation so the call ought to be made somewhere else.

What do you think?

This all sounds pretty reasonable, and I think these suggestions are all worth pitching, at the very least. But you're right — a lot of work here would need to be done in/by Foundation, so it's worth trying to get someone from the team involved to see if they might be interested in this endeavor. (/cc @Tony_Parker)

Regarding PreformattedCodingKey — I think one major question from Codable-type authors will be: "can I get automated CodingKey synthesis but also get the keys to be treated as pre-formatted?" I think the reasonable answer is "no", and I think it's not such a big concern: if your type has strong opinions about how its keys should be handled, it's not unreasonable to need to do a bit of work to indicate that. But, just something to consider, and to spell out in a potential pitch.

1 Like

I agree that automated CodingKey synthesis implies treating the keys as keys rather than 'data' (which is sort of how I see the PreformattedCodingKey: that the keys are data that should be left alone).

I'll add a few other people to this thread: @drexin (since you are reviewing my PR related to SE-0320 which also regards Codable), @tomerd (since you initiated the broader discussion about improvements to Serialization in Swift).

Terms of Service

Privacy Policy

Cookie Policy