Using Unsafe pointers to manipulate a JSON Encoder

Using Swift
1

Hey folks,

I'm working on an issue surrounding evolvability of JSON entities. In order to support this, I need to be able to store and later serialize keys that aren't known by the Swift entity.

For example:

struct V1: Codable {
    var name: String
}

struct V2: Codable {
    var name: String
    var age: Int
}

If a remote endpoint sends V2 to a client that only understands V1, then the unknown properties will be saved and encoded when the entity is sent back over the wire.

As we can't know the types of the unknown values, we can't use the standard process for encoding/decoding (array and dict values aren't necessarily homogenous so we just can't blindly cast)

So I've started thinking about a reflective solution.

I've introduced a protocol that consumers of our library may implement if they care about evolvability:

protocol Evolvable {
    var evolvableUnknowns: [String : Any] { get set }
}

The entities previously referenced would implement this protocol. I then surface two API calls, one for encoding and the other for decoding that they will be responsible for invoking. These calls will do the heavy lifting.

Decoding was simple. Reflect the decoder to get the storage and store all keys that aren't handled by this entity:

let container = try decoder.container(keyedBy: UnknownCodingKey.self)
    var unknownKeyValues = [String: Any]()
    
    let decoderMirror = Mirror(reflecting: decoder)
    let containers = decoderMirror.descendant("storage", "containers")!
    let array = containers as! Array<Any>
    let d = array[0]
    let dict = d as! Dictionary<String,Any>
    
    for key in container.allKeys {
        guard !knownKeys.contains(key.stringValue) else { continue }
        unknownKeyValues[key.stringValue] = dict[key.stringValue]!
    }
    return unknownKeyValues

However, encoding is proving difficult. Since reflection in swift is read-only, I think my only choice is to try using UnsafeMutablePointers.

So the Encoder instance has a private variable called storage which maintains a stack (array) of containers.
I need to get a writeable reference to this stack and replace the value at index 0.

However, the documentation for UnsafeMutablePointers is fairly thin. I'm not quite sure how I can walk the object to get the array and set a new value.

Thoughts?

2

What Encoder and Decoder are you working with here? JSONEncoder provided by Foundation, or your own?

If this is for JSONEncoder/JSONDecoder, manipulating the underlying storage directly is... not supported, at best. If we ever rename the storage and container variables, your code will crash when getting decoderMirror.descendent("storage", "containers")!

A better way to do (until we add something like Unevaluated) would be to define an enum which can represent a JSON hierarchy and to encode and decode self through that:

enum JSON : Codable {
    case null
    case number(NSNumber)
    case string(String)
    case array([JSON])
    case dictionary([String : JSON])

    // define initializers taking the above and recursively converting to JSONs if necessary
    // define init(from:) and encode(to:) to write to a single-value container
}

Using that you can then encode and decode any arbitrary JSON payload and get it back as a strongly-typed JSON enum which you can inspect or vend elsewhere, without the need for accessing private implementation details.

3

If I've understood the question correctly...

Given that the source to JSONEncoder/JSONDecoder is available - which I suspect you probably know already given that you're rummaging around inside its structure - you might be better off just recreating that implementation for now and adding your own hooks.

I followed that approach (copy the code and then hack on it) for DictionaryCoding.

It's obviously not future proof, but it's pragmatic, and you may be able to track any changes made to the original code for now.

I seem to remember that there's a proposal to expose some more of that implementation's internals (which would be useful, not least for what I'm doing in DictionaryCoding). I also think that the implementation is ripe for a bit of refactoring anyway - possible you will end up with something that you can push back as a proposed change (I might, too, at some point...).

4

The proposal I was thinking of was this pitch

5

That's an interesting idea. However, I seem to be somewhat slow on some of the implementation details you're suggesting. This may be due to my being fairly new to Swift.

Would you mind fleshing out the array case as an example?

6

Sure! I actually slapped this together earlier as a gist, though I'll recreate the contents here in case the gist goes away:

import Foundation

public enum JSON : Codable {
    case null
    case number(NSNumber)
    case string(String)
    case array([JSON])
    case dictionary([String : JSON])
    
    public var value: Any? {
        switch self {
        case .null: return nil
        case .number(let number): return number
        case .string(let string): return string
        case .array(let array): return array.map { $0.value }
        case .dictionary(let dictionary): return dictionary.mapValues { $0.value }
        }
    }
    
    public init?(_ value: Any?) {
        guard let value = value else {
            self = .null
            return
        }
        
        if let int = value as? Int {
            self = .number(NSNumber(value: int))
        } else if let double = value as? Double {
            self = .number(NSNumber(value: double))
        } else if let string = value as? String {
            self = .string(string)
        } else if let array = value as? [Any] {
            var mapped = [JSON]()
            for inner in array {
                guard let inner = JSON(inner) else {
                    return nil
                }
                
                mapped.append(inner)
            }
            
            self = .array(mapped)
        } else if let dictionary = value as? [String : Any] {
            var mapped = [String : JSON]()
            for (key, inner) in dictionary {
                guard let inner = JSON(inner) else {
                    return nil
                }
                
                mapped[key] = inner
            }
            
            self = .dictionary(mapped)
        } else {
            return nil
        }
    }
    
    public init(from decoder: Decoder) throws {
        let container = try decoder.singleValueContainer()
        guard !container.decodeNil() else {
            self = .null
            return
        }
        
        if let int = try container.decodeIfMatched(Int.self) {
            self = .number(NSNumber(value: int))
        } else if let double = try container.decodeIfMatched(Double.self) {
            self = .number(NSNumber(value: double))
        } else if let string = try container.decodeIfMatched(String.self) {
            self = .string(string)
        } else if let array = try container.decodeIfMatched([JSON].self) {
            self = .array(array)
        } else if let dictionary = try container.decodeIfMatched([String : JSON].self) {
            self = .dictionary(dictionary)
        } else {
            throw DecodingError.typeMismatch(JSON.self, DecodingError.Context(codingPath: decoder.codingPath, debugDescription: "Unable to decode JSON as any of the possible types."))
        }
    }
    
    public func encode(to encoder: Encoder) throws {
        var container = encoder.singleValueContainer()
        
        switch self {
            case .null: try container.encodeNil()
            case .number(let number):
                if number.objCType.pointee == 0x64 /* 'd' */ {
                    try container.encode(number.doubleValue)
                } else {
                    try container.encode(number.intValue)
                }
            case .string(let string): try container.encode(string)
            case .array(let array): try container.encode(array)
            case .dictionary(let dictionary): try container.encode(dictionary)
        }
    }
}

fileprivate extension SingleValueDecodingContainer {
    func decodeIfMatched<T : Decodable>(_ type: T.Type) throws -> T? {
        do {
            return try self.decode(T.self)
        } catch DecodingError.typeMismatch {
            return nil
        }
    }
}

let json = """
{
    "values": [
        [1, null, "hi"],
        {"hello": "world"}
    ]
}
""".data(using: .utf8)!

let decoder = JSONDecoder()
let decoded = try decoder.decode(JSON.self, from: json)
print(decoded.value)

Error handling here is minimal (and you'd need to expand the .number case) but you can reasonably decode most JSON payloads this way.

You can also combine this with KeyedDecodingContainer.allKeys on a decoding container keyed by CodingKeys which can take on any string/integer value, and with that, iterate through all available keys in the decoding container. Any key you recognize you can decode; any key you don't can be decoded as JSON and stored.

I can expand on this further if it helps.

7

No, this was quite useful, thank you!

1 Like