Revisiting `StructureEncoder` and `StructureDecoder`

Evolution Discussion
1

Continuing the discussion from Idea: Exposing _JSONEncoder and _JSONDecoder functionality:

Back in July, 2017, @itaiferber mentioned the possibility of exposing a StructureEncoder and a StructureDecoder that would work similarly to JSONEncoder and JSONDecoder with the exception that they would not perform the final/initial steps of serializing to and from JSON.

I find this functionality tremendously helpful, so I hope to revive the discussion about this possibility by giving a shot at an implementation.

So, one question I have about swift development in general: My current implementation lives here: GitHub - mortenbekditlevsen/swift at structureencoder
Am I free to create a WIP PR against swift/master even though I haven't discussed this with anyone yet? Or do random PRs like this somehow 'pollute' anything for anyone?

More general questions:

  1. Do anyone else see a benefit in having this Encoder/Decoder-pair?
    My personal use case is the Firebase real-time database from Google. The APIs for storing and retrieving data accept and return Any values containing exactly such JSON-ish data structures.
  2. With regards to the API for this. Does it seem alright to basically duplicate the API for JSONEncoder and JSONDecoder?
    In my oppinion I think that it does, and that it is nice to deal with a familiar API.

Questions that I have encountered in implementing this:

  1. Should a possibly future StructureEncoder and StructureDecoder live in the same file as the corresponding JSONEncoder and JSONDecoder. They share the implementation details (_JSONEncoder and _JSONDecoder which are currently private, but these could of course be made internal instead.
    In the current implementation they do live in the same file.

  2. With regards to the currently private _JSONEncoder and _JSONDecoder: Should they still be named in this way even though they are now the base of two types of encoding/decoding?
    In the current implementation I have not done any renaming.

  3. With regards to the options for StructureEncoder and StructureDecoder, then the options of JSONEncoder and JSONDecoder can be used as they are - with exception of JSONEncoder.OutputFormatting which do not make sense for StructureEncoding.
    The 'cloned' options could either be typealiases for the corresponding JSONEncoder/JSONDecoder options, or they could be completely separate enumerations.
    In the current implementation I have chosen to model them as new enumarations with the exact same structure, and then expose a property that can return the JSONEncoder/JSONDecoder version of the options to be passed on to _JSONEncoder and _JSONDecoder.
    I think that this nicely hides the fact that the implementation details are currently shared.

  4. Are there any assumptions in the _JSONDecoder about the possible values and types that can be present due to the knowledge that the JSONSerialization was the only source of input for the _JSONDecoder.
    I feel that there may be...

  5. For the JSONEncoder, top level fragments are not allowed. Should 'top level values' be allowed in StructureEncoder or not?
    In the current implementation top level values are allowed since I could find no good argument to disallow them.

  6. I have basically cloned all tests from TestJSONEncoder.swift to a new TestStructureEncoder.swift file. I have removed a few tests that relate to formatting, and change the ones that verify that top level fragments are disallowed. For the main workhorse of the test file, I have skipped the equality check of the intermediary value, since the intermediary format is a value of type Any and it's hard to test for equality when one value may be an NSDictionary and the other may be a Dictionary. I think that it is fair to skip this extra test step, since the complete round trip is still tested. Is this an acceptable approach to testing this?

I look forward to seeing any comments on this.

Sincerely,
/morten

5 Likes
Codable with references?
Decoding from dictionary
2

Hi Morten, thanks for continuing the discussion on this! To answer some of your questions:

You technically can, but I think for a PR to be effective, it needs to have a primary purpose. Like for all API changes, the review process requires pitching the API and going through discussion. If you start up a pitch, it'd be great to have a PR up with the associated changes, but that's not strictly necessary; putting up a PR without having an associated pitch, though, might cause some confusion.

Given that this discussion has come up a few times in the past, I think there's been enough demand to pitch it. :slightly_smiling_face:

To answer your more technical questions:

In my mind, if one of the goals in doing this is refactoring some of the implementation we have, one of the easier ways to structure this would be:

  1. Pull out all of the base functionality from _JSONEncoder/_PropertyListEncoder and _JSONDecoder/_PropertyListDecoder into new StructureEncoder and StructureDecoder classes in a new file (StructureEncoder.swift); these classes should be format-agnostic and do the minimum required to get things wrapped up
  2. Because PropertyListEncoder and PropertyListDecoder don't have any encoding strategies, they can use the pure StructureEncoder/StructureDecoder to convert Codable values into containers before passing off to PropertyListSerialization and back. In other words, we can eliminate _PropertyListEncoder and _PropertyListDecoder
  3. Any of the encoding strategies on JSONEncoder/JSONDecoder should remain specific to the JSON format; this means that we would reparent _JSONEncoder/_JSONDecoder to inherit from the new Structure types, and override only the behavior necessary to apply those strategies

Because the top-level {JSON,PropertyList}{En,De}coder types are thin shells around the actual underlying encoding mechanisms, all of the underlying implementation details can change without users knowing; the above is a simple way to do this, but we have flexibility.

There are, yes — for instance, the assumption that all dictionaries must be string-keyed (because that's what JSON supports). In designing a more generalized encoder/decoder pair, we'd need to decide how to handle this:

  1. _JSONEncoder converts Int-keyed dictionaries to String-keyed dictionaries by stringifying the keys. We could maintain this behavior as generally useful: not all formats support Int-keyed dictionaries, so it would be a shame to need to subclass these types just to enforce that. Alternatively, we could offer an encoding strategy for it to let consumers decide
  2. Dictionary itself already turns non-String- and non-Int-keyed dictionaries into keyed containers, so we wouldn't need to handle more complex cases than that

Of course, there are other potential implicit assumptions that we'd need to audit for.

Agreed — there's no need for the application of JSON semantics to affect this more generalized structure. (In general, too, there have been requests to relax this restriction, which I'm in favor of; that will require further changes, though.)

Thanks for putting work into this! We can continue discussing specifics, but I'd like to pull back to some higher-level topics here that in the past we haven't gotten to come up with good answers for. Specifically, two subjects:

  1. Naming/availability
  2. Whether or not JSONEncoder/JSONDecoder will use this new pair as their basis

The first subject is the much bigger and more important one: at the moment, the namespace here is already pretty saturated with "Encoder"- and "Decoder"-type words that can make reaching for the right tool to use a little bit difficult. Besides Encoder and Decoder themselves, there are the encoding containers, and the actual format-specific encoder types. If we think this is something we'd like to do, we need to come up with a really good name for these types that

  1. Doesn't have the potential for leading toward erroneous conflation of "Structure" with struct, i.e., we don't want to run the risk of someone reaching for StructureEncoder for the wrong reason
  2. In general, is more difficult to reach for to begin with. These types are not necessarily all that useful on their own; it's rare to need to need to convert arbitrary Codable types to containers unless you're then going to pass those containers through a serialization pass of your own. This makes them good API for developers who are looking to write their own encoders and decoders, but not so good for developers looking to use encoders and decoders. StructureEncoder is currently a straw-man name to give a bit of meaning to the API, but something like ConcreteEncoderBase (or something similarly abstract) could make it more difficult to reach for

Something that has come up in the past is potentially namespacing these types to begin with to fall more in line with the principle of progressive disclosure. However, in the absence of language-supported namespaces at the moment, we'd have to resort to using something like enum EncodingUtilities {...}, which would still be imported by default when you import Foundation. (I think ideally, you'd need to import Foundation.EncodingUtilities or similar in order to see these types.) Not something insurmountable, but not quite in place yet for us to be able to use it.

As for JSONEncoder/JSONDecoder — while it would be really nice to factor some of this implementation out into something shareable between JSON and PropertyList, another topic that's come up is weaning JSONEncoder and JSONDecoder off of JSONSerialization and onto a different internal serializer that's closer to the whole encoding and decoding process. Not requiring a whole pass to decode the JSON via JSONSerialization first would give us better type-level access to the underlying data (e.g. see decoding a Double vs. decoding a Decimal; JSONSerialization has to choose which, but doesn't know what you'd prefer as a consumer — while JSONDecoder might and could do a better job), along with some potential performance benefits.

Does that obviate the need for this? I don't think so — clearly this would be useful on its own. The question is just how much infrastructure work would we like to do to factor this out while then repeating further work to further enhance JSONEncoder/JSONDecoder. I don't have an answer for this at the moment.

In any case, I appreciate the continuing discussion! I think before we skip ahead to implementation detail, we should carefully consider the design decisions that make a big difference here, then pitch the API. The implementation detail should be "boring" in comparison to the work done up-front. :slight_smile:

2 Likes
3

Hi @itaiferber,
Thank you for your comments.

I realize that my own focus on exposing the JSONEncoder and JSONDecoder internals are too narrowly focused - and 100% driven by my specific use case, which may not be a benefit to the broader swift community.

My own use case is that I would like the exact behavior of JSONEncoder and JSONDecoder, but just without the JSONSerialization steps for serializing and deserializing to and from Data.

Considering your comments that something like a ConcreteEncoderBase or StructureEncoder is specifically not for encoding JSON and thus should not have similar key coding strategies and number overflow handling, I realize that what I am looking for is perhaps something different than a ConcreteEncoderBase.

I hope that I do not twist the purpose of this thread immediately after starting the discussion by airing this other thought:
I now realize that the Encoder and Decoder I need is still in the realm of JSON - but I would just like another representation than pure Data.

Today JSONEncoder and JSONDecoder converts between Codable models and Data.
It has been mentioned, that a streaming API would be a really nice thing to have too at some point in the future.

If such APIs were added, would these be overloads of the encode and decode functions - or perhaps new functions added to the classes?

If that is the case, would it make sense to provide similar overloads or new functions to JSONEncoder and JSONDecoder that converts to and from Dictionary/Array/value structures (typed as Any)?
One could even imagine a similar overload for converting to and from String representations of the data too?

For instance:

// Current API
let encoder = JSONEncoder()
let data = try encoder.encode(pear)

// Streaming API
let stream = try encoder.encodeToStream(pear)

// String API
let jsonString = try encoder.encodeToString(pear)

// Key/Value structure API
let structure = try encoder.encodeToStructure(pear)

Ok, sorry for derailing the discussion a bit.

With regards to a ConcreteEncoderBase I agree that namespacing more low-level functionality would be optimal. And namespacing by using an enum might not be a good idea since this decision will have to live on forever due to ABI stability.
Does this suggest that perhaps it would be better to wait with exposing a ConcreteEncoderBase until language support for namespaces is added (assuming that this is something that will happen even though I haven't followed any discussions on this).

4

No worries — this is a totally valid use-case! Unfortunately, though, I don't know just how generally useful converting a Codable tree to something that would be JSON once serialized is, and whether it makes for good API. We do have something like this on PropertyListEncoder, but it's for internal use only in NSKeyedArchiver.

My best answer for you at the moment is that Swift is licensed under the permissive Apache; as long as you follow the legal requirements, you can always grab a copy of JSONEncoder/JSONDecoder and modify it to fit your needs, and others have done this in the past. Unfortunately, the hard part is then keeping your changes in sync with changes on our end, so it's not a truly perfect solution.

Unfortunately, all I can say at this point is "I don't know". It's not clear yet what a streaming solution would look like, and what kind of streaming we'd want to do. Foundation offers the Stream type and subclasses (and JSONSerialization can stream to/from them), but folks very rarely use these types. We'd need to put in some thought into what this would look like; with a stream, it's conceivable that we'd want the API to be asynchronous, too, rather than synchronously waiting for all the data to be written to the stream.

As for encoding directly to a structure: as far as API surface goes, I think we'd largely prefer to promote writing to JSON, since that's JSONEncoder's purpose. Clearly there are some use cases for it, and implementation difficulty is not a factor at all, but does it make good API? This is something that requires some thought.

Agreed — I don't think now is a good time to make such a long-lasting decision, especially since there is a lot of discussion going around as to what namespaces and submodules might look like.

And yes, this is largely why we haven't had a great answer for this yet — it's definitely nice to have and is somewhere on a potential roadmap; implementing this is not the difficult part: it's the API design and naming that we have to work on, since those live forever.

Once Swift has a solution to namespacing in place, whatever shape that takes, this'll be much easier to move forward with.

1 Like
5

No problem, that's exactly what we are already doing :slight_smile: (including proper entitlements). It's not too bad to keep the changes in sync. Starting over from a new version of JSONEncoder.swift takes less than half an hour. :-)