How to deal with completely dynamic JSON responses

Using Swift
1

Maybe someone in the community has had similar struggles and have come up with a workable solution.

We're currently working on a polyglot key/value store. Given this, we'll generally have no knowledge of what will be stored ahead of time.

Consider the following struct

struct Character : Codable, Equatable {
    let name:    String
    let age:     Int
    let gender:  Gender
    let hobbies: [String]
    
    static func ==(lhs: Character, rhs: Character) -> Bool {
        return (lhs.name == rhs.name
                   && lhs.age == rhs.age
                   && lhs.gender == rhs.gender
                   && lhs.hobbies == rhs.hobbies)
    }
}

When sending/receiving Character entities over the wire, everything is fairly straight forward. The user can provide us the Type in which we can decode into.

However, we do have the ability to dynamically query the entities stored within the backend. For example, we can request the value of the 'name' property and have that returned.

This dynamism is a pain point. In addition to not knowing the type of the properties outside of the fact that they are Codable, the format that is returned can be dynamic as well.

Here's some examples of response for two different calls extracting properties:

{"value":"Bilbo"}

and

{"value":["[Ljava.lang.Object;",["Bilbo",111]]}

In some cases, it could be an equivalent of a dictionary.

Right now, I have the following structs for dealing with responses:

fileprivate struct ScalarValue<T: Decodable> : Decodable {
    var value: T?
}

Using the Character example, the type passed to the decoder would be:

ScalarValue<Character>.self

However, for the single value, array, or dictionary case, I'm somewhat stuck.

I've started with something like:

fileprivate struct AnyDecodable: Decodable {
    init(from decoder: Decoder) throws {
        // ???
    }
}

Based on the possible return types I've described above, I'm not sure if this is possible with the current API.

Thoughts?

2

/cc @itaiferber

3

I think you would need to encode the name of the type (or another value you can map to a type) along side the encoded value. After you have a type you could cast it to Decodable and then decode an instance. Unfortunately Swift doesn't have a way to lookup a type based on a string yet but there has been a lot of progress in that area recently. In the meantime, you would need to maintain a lookup table manually. That table could be [String: Decodable] which would allow you to avoid the cast.

4

What are you looking to do with the end result of the decoded value? Are you passing that along to another API, or looking to consume that somehow yourself?

JSONEncoder and JSONDecoder currently don't have an answer for totally arbitrary input; one of the core components of Codable is requiring you to know what type you're looking to decode and asking for that. If you're looking to decode one of multiple known types, then Swift's existing solution to a multiple type representation is the way to go — making a Codable enum representing those types, e.g.:

enum StringOrInt : Codable {
    case string(String)
    case int(Int)

    init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        do {
            self = .string(try container.decode(String.self))
        } catch DecodingError.typeMismatch {
            self = .int(try container.decode(Int.self))
        }
    }

    func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        switch self {
        case .string(let string): try container.encode(string)
        case .int(let int): try container.encode(int)
        }
    }
}

It's similarly possible to write an enum which can represent all of the underlying JSON types and encoding and decoding that. It's a bit more verbose, but possible.

I've also been interested in adding JSONEncoder.UnevaluatedJSON and JSONDecoder.UnevaluatedJSON, special wrappers around Any which allow you to encode and decode otherwise untouched JSON types (Dictionary, Array, Int, String, etc.). These would be handled specifically by JSONEncoder and JSONDecoder to essentially leave whatever types they contain alone. So, for instance, you'd write

let container = try decoder.singleValueContainer()
self.stuff = try decoder.decode(JSONDecoder.UnevaluatedJSON)
// self.stuff.value now contains whatever was in the container, e.g. Int or String or [String : [Double]].

Of course, it doesn't make sense to ask a different decoder (e.g. PropertyListDecoder) to decode JSONDecoder.UnevaluatedJSON, so this concept would need to be further refined as API. I don't think we'll have time to do this in the Swift 5 timeframe.

In any case, more information about the specifics of what you're trying to do with that polymorphic value would help give a more pointed solution.

5

Note that we don't recommend attempting to decode values from arbitrary payloads by reading a string from the payload and converting that into a type at runtime using a Swift equivalent of NSClassFromString, if it comes in the future. In general, data like that is untrusted input, and without gating the types that you could instantiate from such input, you could easily run an attacker's code execution exploit.

This is why we have (and recommend you use) NSSecureCoding in ObjC, to prevent this sort of abuse.

The best way to do this and keep it secure is maintain an immutable lookup table of types you're willing to decode, either via an actual table, or better yet, with an enum as recommended above.

1 Like
6

I had to write a JSONValue type like this recently to handle truly arbitrary JSON values. It's really not that hard to do and it integrates cleanly with other types in contexts where you know more about the surrounding structure.

1 Like
7

Yep, in general, this would be the way to go. If Swift gets variadic generics at some point, we might be able to introduce enum OneOf<T...> : Codable where T... : Codable (or similar) to cut down on the boilerplate, but that's neither here nor there at the moment. :)

8

Thanks for chiming in with the caution about security! Just because it's possible doesn't mean it's a good idea! :)

3 Likes
9

I was just getting this linked by a colleague. I think OneOf<T...> would be a brilliant use case for some form of variadic generics, of possible.