Decode a JSON object of unknown format into a Dictionary with Decodable in Swift 4

[jrose (Jordan Rose)]

@itaiferber

How does adding typealiases to protocols interact with the ABI?

Typealiases themselves are never part of the ABI, but they are an important part of API, so adding them without thought can cause plenty of problems. It does look like we have existing lookup / overload resolution bugs today—can you file a bug for the example you just made up?

The more general thing about being able to have a "constrained protocol composition" is actually a simplified form of "generalized existentials". I don't know if we've ever talked about having that capability first.

[itaiferber]

@jrose

Typealiases themselves are never part of the ABI, but they are an important part of API, so adding them without thought can cause plenty of problems.

Okay, that's what I remembered; typealiases are not ABI-related. Of course, introducing new typealiases adds API which would break anyone currently already extending these protocols with the same names.

It does look like we have existing lookup / overload resolution bugs today—can you file a bug for the example you just made up?

Sure! Filed as SR-7217.

[anandabits (Matthew Johnson)]

@itaiferber

If you’re aware of the structure of the JSON you’re looking at, is there anything preventing you from casting the .value to the structure you want? (Say, [String : [Any] for a dictionary containing arrays of heterogeneous values)

I am having trouble thinking of a case where I would prefer to use Unevaluated as opposed to a more concrete type with a relatively dynamic (but bounded) structure such as something like this:

enum DynamicValue {
    case none
    case bool(Bool)
    case string(String)
    case double(Double)
    case dictionary([String: DynamicValue])
    case array([DynamicValue])
}

Are there cases where a type like this would not work but Unevaluated would work?

[cherrywoods]

The RawJSON type I suggested above could look like this. I think you would only need to use Unknown when you have more than one format you want to decode from.

This is what I ment with

@cherrywoods

There is no indication that there is just a limited set of options, but there is actually just a limited set of possible types for this Any value, because it is JSON, as we found out.

Your example explains way better what I mean (Thanks )

[itaiferber]

@cherrywoods

I do not like the idea of such an abstract Unevaluated type, although I think it is in general the right way to solve the problem discussed in this thread.

I think it would be better to add something like this specifically for JSON. I furthermore think this could be a great place to provide some more functionality for accessing "raw" JSON by making Unevaluated more than just a marker type.

Let's pull this apart a little bit, if you don't mind, because I think there's been a bit of correlation here between proposals that I'd like to separate. I'm going to partially reorder your comment to answer sequentially.

Im sticking a bit too much to the implementation example for above. Beyond this, an abstract implementation would leave us with those problems:

• There is no indication that there is just a limited set of options, but there is actually just a limited set of possible types for this Any value, because it is JSON, as we found out.
• What is the decoders internal representation if it isn't JSON?

The original request for Unevaluated, both partially here, and in SR-5311 is an answer to the currently intractable case of "there is a part of my serialized object graph that I know nothing about, and need to be able to decode without knowing anything about it, for the sake of preserving the values as-is." Usually this comes up when you are decoding parts of a payload whose structure is variable — you need some of the information from it, but not all, and the parts you don't need (and know nothing about) need to be preserved for re-encoding back later on.

The idea here behind Unevaluated is to give you that data, as-is, in a format that may or may not be opaque to you. You can later re-encode the Unevaluated instance, and ideally, get back the same representation that you had before.

This is separate from a request for returning a parsed response in a reasonably consumable format.

The problem I see, is this: You somehow find out, that the format you are currently decoding from is JSON. Then you decode such an Unevaluated value from the decoder. Now you have to do something with either the NSDictionary, or NSArray, or whatever you get as .value. Those types are as I see it an implementation detail of JSONDecoder (O and what you get with Unevalutuated is actually a "raw" view into the decoders storage. This has in my opinion several disadvantages:

• It would be just annoying to refactor the custom decoding code, if JSONDecoder started to use Dictionary instead of NSDictionary, Array instead of NSArray, and so on.

This shouldn't be problematic as those types bridge.

• The resulting code won't be easily understandable I guess, because you won't be able to figure out right away, that this code is working with "raw" JSON / I would not immediately connect NSDictionary with JSON's "{ }". At least it won't be self documenting.

This is also no different from manually working with a response from JSONSerialization. You would be getting back the same response, wrapped up in an Unevaluated.

Also, I do dislike the concept of an abstract type for all formats, because I fell like it does not match the design of Decoder. I would consider it better to have some method like abstractRepresentation alongside with keyedContainer, unkeyedContainer and singleValueContainer. The problem with this is, that the return type is still open...

As you say, what would the result of .abstractRepresentation be? How would it be different from the abstract representation contained in an Unevaluated?

If we had a type such as RawJSON (I'm really not good at naming), we could add general working-with-JSON capabilities and not show the NSDictionarys under our cloths to the world. One could ask this type whether it has a keyed container for us, or a number, and so on. We could also tell it that we want to have a number, also if there is a string (only if it is convertible of course). These examples are certainly not well designed right now, but I think it shows what I mean. This new type would could take a way lager role in the JSON decoding game. We could start with such a RawJSON, do some lookup and extraction (like resolving the upper container levels that maybe contain information we don't really want to have as a type on it's own, I don't know if this is happens often, but I have seen something similar with another format) and then at some other point, tell JSONDecoder to decode the remaining JSON for us. If we need to handle something like the meta data from this issue, we could go on manually working with the JSON and if it contained other data, that would rather like to have as an instance of one of our custom types, we could just go on as we started: We would tell JSONDecoder to decode the remaining RawJSON extracted before.

I don't think wo should miss this chance with adding an abstract type. Other implementations for other formats could follow the same way, but would not be required to do so.

This, to me, sounds like the real request here — working with a concrete result type closer to the JSON type I gave an example for in another thread or that @anandabits just posted as well. [This enum is possible today, and relatively easy to put together.]

However, again, I don't think we want to offer types specifically for JSON (especially when they're relatively easy to put together without special internal knowledge). See part of my response upthread to @hooman:

@itaiferber

When designing Codable, one of the big decisions we made early on was that we wanted Codable to be as format-agnostic as we could make it (without making it so abstract as to no longer be helpful); our goal was to help abstract away just enough details to make a given Codable implementation work with many different formats. As such, many existing Codable implementations would work as-is whether encoded through JSONEncoder, PropertyListEncoder, or any other Encoder you might write.

With this, I recognize that the #1 use-case by far for Codable at the moment is for serializing to and from JSON; it's a really popular format at the moment, and when most folks write their Codable implementations, they've got JSON in mind.

<snip>

In any case, our philosophy on this is generally that if JSON can benefit from such a feature, other formats should be able to benefit as well. We'd rather not gate this on JSON adoption specifically, especially as we're looking to add new encoders and decoders ourselves to support further formats.

If you need a JSON enum like above, it's possible to write it yourself (and please feel free to use my gist as reference); however, you cannot today write Unevaluated on your own as it requires explicit adoption by JSONEncoder/JSONDecoder.

[anandabits (Matthew Johnson)]

Right, I have a type like this called JSONValue but since this conversation is about supporting multiple formats I shared it as DynamicValue. The main point is that this type's implementation of Codable need not be tied to any specific encoder or decoder. It would work with any implementation that supported the necessary structure.

What I am having trouble imagining is a case where you would want to work with an Unevaluated value without knowing anything about its structure. You could get the unevaluated value and use a bunch of casts or you could just use a type that describes the structure you expect. The latter seems like a better approach to me, at least in most cases. Are there counter-examples that I am not considering?

[cherrywoods]

Thanks for your detailed answer.

My answer indeed included multiple requests, one of wich is a better overlay for JSONSerialization, I guess.

@itaiferber

The original request for Unevaluated, both partially here, and in SR-5311 is an answer to the currently intractable case of “there is a part of my serialized object graph that I know nothing about, and need to be able to decode without knowing anything about it, for the sake of preserving the values as-is.” Usually this comes up when you are decoding parts of a payload whose structure is variable — you need some of the information from it, but not all, and the parts you don’t need (and know nothing about) need to be preserved for re-encoding back later on.

I wasn‘t thinking about this. In this case it totaly makes sence to me to have such an marker. Would it be possible to unify both concepts and use a Unevalutated protocol and a concrete JSON type? .abstractRepresentation would then return Unevaluated in general and JSON in the JSON implementation.

I‘m not realy informed about all this, I do not even work much with JSON. The workflow I discribed came to my mind when working with msgpack and adding Codable support to an existing framework.
I thought something like JSON could solve or help solve many of the problems described here, just by providing more usefull functionality. Whether this is true or not needs to be verified.

[anandabits (Matthew Johnson)]

@itaiferber

The idea here behind Unevaluated is to give you that data, as-is, in a format that may or may not be opaque to you. You can later re-encode the Unevaluated instance, and ideally, get back the same representation that you had before.

Thanks for elaborating. I can imagine there might be case where you want to be able to round-trip an opaque value and a concrete dynamic structure would not be able to do that.

@itaiferber

However, again, I don't think we want to offer types specifically for JSON (especially when they're relatively easy to put together without special internal knowledge).

I am also skeptical of this in general. The specific dynamic structure that could be useful really depends on the specific circumstances. The standard library should not speculate about these as they may change over time. If they did get added I think they would make most sense in Foundation, along side the encoders and decoders.

That said, it isn't entirely uncommon to encounter the need for a type to support dynamically structured data in popular formats. By definition these types can only be compatible with encoders and decoders that support the structure they require. Perhaps it is worth including a high quality and shared implementation of popular dynamic structures like JSONValue and PropertyListValue along side the encoders and decoders. These could correspond to the Unevaluated type for the corresponding encoder and decoder if that feature was also added.

[cherrywoods]

To come back to your original question, I think the best way right now is just to use some JSON enum, similar to the one from itaiferber's gist he referred to earlier in this thread. You can use the implementations of init(from:) and encode(to:) in this gist to extend e.g. Argo's JSON enum to conform to Codable by just changing two words (.dictionary to .object). If you have such an enum, you would decode metadata as [String:JSON].

Depending on what you want to do with this metadata later, you could also use a new meta data structure or enum, whose init(from:) method could look like this:

init(from decoder: Decoder) throws {
    let container = decoder.singleValueContainer()
    let meta = try container.decode([String:JSON].self)

    // e.g. look for keys in meta and then chose for example a case of this enum
}

This is just a summary of the previous answers I think, but in my understanding it should be enough to solve your issue.

[cherrywoods]

@itaiferber

We can certainly add some dynamic lookup features to Unevaluated to make it more accessible, and I think that would be a great extension to the feature (and a good use-case for dynamic lookup)! However, I think we’d rather avoid adding JSON-specific types for this.

===========================================================
EDIT: This is deprecated, it is rewritten in anther topic, see next post from me.

I'm not sure whether I understand dynamic lookup right. If I do, won't Unevaluated just be another decoder?

I mean: As abstract type, would we ask Unevaluated whether it is e.g. a keyed container?
I think then it would be just a second decoder.

If we would not: would it be an enum similar to DynamicValue, like this:

enum Unevaluated {
    case keyedContainer([String:Unevaluated])
    case unkeyedContainer([Unevaluated])
    case singleValue(Any?)
}

I don't think this is good, because it isn't extendible. For example in msgpack you can have arbitrarily keyed containers and it is reasonable and possible to provide this feature as a method like keyedContainer on the encoder, decoder and the keyed and unkeyed containers, but it was not possible to add it to Unevaluated, because it needed another case.
I don't think that there should be a restriction to only keyed, unkeyed and single value containers.

However to add dynamic lookups beyond what decoder can do, I think, we would need to assert more about the serialization formats (which I do not support).

Maybe I‘m just missunderstanding „dynamic lookup“.

For an abstract Unevaluated struct just for re-encoding like this one:

@itaiferber

public struct Unevaluated : Codable {
    public let value: Any
    public init(_ value: Any) { self.value = value }

    public init(from decoder: Decoder) throws {
        // throw a type mismatch
    }

    public func encode(to encoder: Encoder) throws {
        // throws an invalid value error
    }
}

I see just a minor issue:
The abstract concept somehow suggests to me that I can also use this with other encoders than the one related to the decoder I got the unevaluated from. I should of course not do this, but I could mix JSONDecoder and PlistEncoder and succeed in doing so, if the JSON only contained Dictionary, Arrays, Strings and Numbers and the implementation is this one:

@itaiferber

fileprivate func unbox<T : Decodable>(_ value: Any, as type: T.Type) throws -> T? {
    if type == Unevaluated.self {
        return Unevaluated(value)

After all, I'd rather liked to have a protocol called Unevaluated then a struct.

With a protocol you could also connect this with format specific lookup and manipulation support, e.g. with such a JSON enum that conforms to Unevaluated. If passing this to e.g. a PlistEncoder, one would get a clear error here, or JSON could even support such a cross over by implementing Encodable and encoding the way it is implemented in itaiferber's gist. Unevaluated could be documented as a good point to implement format specific dynamic lookups and manipulation.

One disadvantage of a protocol is that it won't be possible to call decode(Unevaluated.self) to get raw storage of a decoder to encode it later, as far as I can see. A method returning Unevaluated would work, but this would require all decoders to support it. However all decoders should have some sort of data they are working that they can pass back here.

================================================================
This is still a bit relevant, but maybe not.

Beyond this, I think it would be good to add a JSON enum to Foundation. It could be provided as an overlay for JSONSerialization. It could also replace JSONEncoder and JSONDecoder in general (although this might not be desirable) (whose names are a bit confusing in correlation to Encoder and Decoder) by providing something like this:

struct JSON {
    
    /// the internal representation of the JSON 
    private var parsedJSON: Any
    // if using JSONSerialization, this any would be eigther NSNull, NSNumber, NSString, NSArray or NSDictionary

    public var options // TODO: options

    // MARK: encoding
    init<E: Encodable>(encoding value: E) throws {
        let encoder = _JSONEncoder(options: self.options)

        guard let topLevel = try encoder.box_(value) else {
            throw EncodingError.invalidValue(value, EncodingError.Context(codingPath: [], debugDescription: "Top-level \(T.self) did not encode any values."))
        }
        // creating custom representation from result and setting parsedJSON
        // _JSONEncoder could also use JSON directly
    }

    var dataValue: Data {
       return JSONSerialization.data(with: parsedJSON) // + options
    }

    // MARK: decoding
    init(decoding data: Data) throws {
        // calling JSONSerialization.jsonObject
        self.parsedJSON = JSONSerialization.jsonObject(with: data) // + options
    }
   
    func decode<D: Decodable>(to type: D.Type) throws -> D {
        let decoder = _JSONDecoder(referencing: parsedJSON, options: self.options)
        guard let value = try decoder.unbox(topLevel, as: type) else {
            throw DecodingError.valueNotFound(type, DecodingError.Context(codingPath: [], debugDescription: "The given data did not contain a top-level value."))
        }

        return value
    }
}

Again, these are two diffrent things in one post. I will create a new topic for the JSON enum later. I will come back to this, if it can not be solved with dynamic lookup on Unevaluated.

[dlbuckley (Dale Buckley)]

Considering how much of this topic is getting confused with other concepts I'll refrain from adding more here. I'll bring them up in a different thread or somewhere else.

[itaiferber]

I don’t have time at the moment to put together a full response to your comments (hopefully later or tomorrow), but the dynamic lookup I’m referring to is the Dynamic Member Lookup introduced by SE-0195. The idea would be to add some dynamic members to Unevaluated to make it somewhat consumable without needing to cast its .value to something concrete.

[cherrywoods]

Thanks for this hint! I misunderstood dynamic lookup. I will summand what I wanted to say in another post, in a new topic: New "Unevaluated" type for Decoder to allow later re-encoding of data with unknown structure (because I think it is no longer really related to this question)