AnyCodable Efficacy

Hello, Swift community!

I was playing around with Codable type-erasure types and realized that Codable doesn't have a proper type-erasure container, e.g. AnyCodable.

So I played around with the idea and came up with an implementation that encodes type information and the wrapped Codable value and then recovers the type to instantiate the Codable value.

struct AnyCodable: Codable {
   let box: Codable

   func encode(to encoder: Encoder) throws {
        var container = encoder.unkeyedContainer()
    
        try container.encode(
            AnyCodableType(type(of: box))
        )
        try box.proxyEncode(to: &container)
    }

    init(from decoder: Decoder) throws {
        var container = try decoder.unkeyedContainer()

        let type = try container.decode(AnyCodableType.self).typeBox
        box = try type.proxyDecode(from: &container)
    }
}

extension Encodable {
    /// Used to call `container.encode(_:)` with a value bound
    /// to the `Encodable` existential as the first parameter.
   func proxyEncode(to container: inout UnkeyedEncodingContainer) throws {
        try container.encode(self)
    }
}

extension Decodable {
    /// Used to call `container.decode(_:)` with a value bound
    /// to the metatype of the `Decodable` existential as the first parameter.
   static func proxyDecode(
        from container: inout UnkeyedDecodingContainer
    ) throws -> Self {
        try container.decode(self)
    }
}

Here, the AnyCodableType is a Codable wrapper for metatypes of Codable-conforming types.

struct AnyCodableType: Codable {
    typealias MemoryLayout = (UInt64, UInt64, UInt64)

    let typeBox: Codable.Type
    
    init(_ typeBox: Codable.Type) {
        self.typeBox = typeBox
    }

    func encode(to encoder: Encoder) throws {
        let memoryLayout = unsafeBitCast(
            typeBox,
            to: MemoryLayout.self
        )
        
        try memoryLayout.encode(to: encoder)
    }
    
    init(from decoder: Decoder) throws {
        let memoryLayout = try MemoryLayout(from: decoder)
        
        typeBox = unsafeBitCast(
            memoryLayout,
            to: Codable.Type.self
        )
    }
}

I tried this on a playground and it worked :tada:. I don’t know, however, if there are any pitfalls in terms of performance and correctness. Furthermore, if I haven't made a major oversight, would it be production-grade or would it be confined to pet projects?

I'd love to hear your opinions on this implementation and to learn about other attempts at creating a Codable type-erasure container!

4 Likes

Can you explain what guarantees in the ABI that it relies on? I would have assumed that encoding/decoding the bits of the metatype isn’t something you can do reliably, but I don’t know much about the ABI.

1 Like

Yeah, this, uh, only works within a single run of an app. The bits of the metatype value are an address, and that address may be different next time the program is run.

5 Likes

Here's my attempt which I built to handle coding Void in my async network stack.

fileprivate struct CodingProxy<Wrapped> {
    let value: Wrapped
}

extension CodingProxy: Codable {
    init(from decoder: Decoder) throws {
        if let type = Wrapped.self as? Decodable.Type {
            value = try type.init(from: decoder) as! Wrapped
        } else {
            fatalError()
        }
    }
    
    func encode(to encoder: Encoder) throws {
        if let encodable = value as? Encodable {
            try encodable.encode(to: encoder)
        } else {
            fatalError()
        }
    }
}

extension JSONEncoder {
    func encode<T>(_ value: T, fallback: (Any) throws -> Data) throws -> Data {
        if T.self is Encodable.Type {
            return try encode(CodingProxy(value: value))
        } else {
            return try fallback(value)
        }
    }
}

extension JSONDecoder {
    func decode<T>(_ type: T.Type, from data: Data, fallback: (Data) throws -> Any) throws -> T {
        if T.self is Decodable.Type {
            return try JSONDecoder().decode(CodingProxy<T>.self, from: data).value
        } else {
            return try fallback(data) as! T
        }
    }
}
1 Like

Nice opaque wrapper for codable!

One change I'd make is to rely on and preserve Swift's safety. For example, JSONDecoder.decode may unexpectedly crash if fallback returns a type that is not T. So I'd recommend requiring that the fallback return T, since this method appears to be a general-purpose API.

Through some nifty generic tricks, you could also remove the fatal errors on CodingProxy to raise any implementation issues to compile time.

Overall, good job!

Filip

1 Like

The issues you mentioned are the exact one's a went back and forth on so and extra thank you for you comments.

The fatalError() calls should never happen so I was happy for them to cause a crash. I would love to know what niffy tricks I could use to engineer them out as I've come to the conclusion that types and type inference is the real Swift language with for loops etc being a thing you can get anywhere.

Regarding the decoding fallback returning Any, both systems worked so I could have thrown an error if the type wasn't right (same with the fatal errors). I decided in the end to match the encoding fallback for aesthetics.

1 Like

OK, this is what I've got:

// The existential box is opened on `Encodable` methods, 
// so we can get access to `Self: Encodable` instead of `Encodable`.
extension Encodable {
    func encode(to jsonEncoder: JSONEncoder) throws -> Data {
        try jsonEncoder.encode(self)
    }
}

extension JSONEncoder {
    func encode<T>(_ value: T, fallback: (Any) throws -> Data) throws -> Data {
        guard let value = value as? Encodable else {
            return try fallback(value)
        }
        
        // Just relay to our existential-opening proxy method.
        return try value.encode(to: self)
    }
}

extension Decodable {
    // The same as for `Encodable`.
    static func decode(to jsonDecoder: JSONDecoder, data: Data) throws -> Self {
        try jsonDecoder.decode(self, from: data)
    }
}

extension JSONDecoder {
    func decode<T>(_ type: T.Type, from data: Data, fallback: (Data) throws -> T) throws -> T {
        guard let decodableType = type as? Decodable.Type else {
            return try fallback(data)
        }
        
        // Unfortunately, we do have to do an unsafe cast, but it's
        // all in one place. We do the cast, get a `Decodable` box,
        // operate on it and then cast in *one* function.
        return try decodableType.decode(to: self, data: data) as! T
    }
}

As I mentioned the above approach still uses an unsafe cast, which I haven't found a way to get rid of without SE-0309. So until Swift 5.7 we will have to bear an unsafe cast, but in the above case, the type erasure and casting happen in close proximity and are thus less error-prone.

1 Like

Simply exquisite! Existential unboxing has been my bane for a long time; I should have known the solution was pushing forwards through extensions on a protocol.

1 Like

how do you improve this with swift 5.7 now available?

It becomes more interesting if you need to support the Sendable protocol for Codable objects ))

I tried this last night and thought it was interesting. Although I am not sure if I captured the original idea correctly or not. But maybe?

enum CodingProxyError: Error {
    case failedDecoding
}

fileprivate struct CodingProxy<Wrapped> {
    let value: Wrapped
}

extension CodingProxy: Codable {
    init(from decoder: Decoder) throws {
        if let type = Wrapped.self as? Decodable.Type {
            do {
                if let wrappedValue = try type.init(from: decoder) as? Wrapped {
                    value = wrappedValue
                    return
                }
            }
        }
        throw CodingProxyError.failedDecoding
    }

    func encode(to encoder: Encoder) throws {
        if let encodable = value as? Encodable {
            // no shell was needed
            try encodable.encode(to: encoder)
        } else {
            // shell was needed
            try [Self]().encode(to: encoder)
        }
    }
}

fileprivate struct AnyCodingProxyB<T:Any>: Codable {}
fileprivate struct InvertedCodingProxyC<T:Any & Codable>: Codable {
    let value: T
    init(from decoder: Decoder) throws {
        if let type = T.self as? Decodable.Type {
            do {
                if let wrappedValue = try type.init(from: decoder) as? T {
                    value = wrappedValue
                    return
                }
            }
        }
        throw CodingProxyError.failedDecoding
    }

    func encode(to encoder: Encoder) throws {
        if let encodable = value as? Encodable {
            // no shell was needed
            try encodable.encode(to: encoder)
        } else {
            // shell was needed
            try [Self]().encode(to: encoder)
        }
    }
}

final class AppTests: XCTestCase {

    func test_codability() throws {
        let encoder = JSONEncoder()
        let withObjectOfCodableInteriorType = CodingProxy(value: "my codable type")
        let dataFromCodable = try encoder.encode(withObjectOfCodableInteriorType)
        let withObjectOfNonCodableInteriorType = CodingProxy(value: Void.self)
        let dataFromNonCodable = try encoder.encode(withObjectOfNonCodableInteriorType)
        let decoder = JSONDecoder()

        // decoding does not yield a nested shell
        XCTAssertThrowsError(try decoder.decode([Data].self, from: dataFromCodable))

        // decoding yields a nested shell
        XCTAssertNoThrow(try decoder.decode([Data].self, from: dataFromNonCodable))

        // checking if the first observation holds for the whole big type
        XCTAssertThrowsError(try decoder.decode(CodingProxy<[Data]>.self, from: dataFromCodable))

        // checking if the second observation holds for the whole big type
        XCTAssertNoThrow(try decoder.decode(CodingProxy<[Data]>.self, from: dataFromNonCodable))

        // checking if the first observation holds when the interior type isn't already nested
        XCTAssertThrowsError(try decoder.decode(CodingProxy<Data>.self, from: dataFromCodable))

        // checking if the second observation holds when the interior type isn't already nested
        XCTAssertNoThrow(try decoder.decode(CodingProxy<Data>.self, from: dataFromNonCodable))

        // we don't need the erased type to determine that it was codable
        XCTAssertThrowsError(try decoder.decode(InvertedCodingProxyC<[Data]>.self, from: dataFromCodable))

        // we don't need the erased type to determine that it was not codable
        XCTAssertNoThrow(try decoder.decode(InvertedCodingProxyC<[Data]>.self, from: dataFromNonCodable))

        // some extra observations
        XCTAssertNoThrow(try decoder.decode(AnyCodingProxyB<Any>.self, from: dataFromCodable))

        XCTAssertNoThrow(try decoder.decode(AnyCodingProxyB<Any>.self, from: dataFromNonCodable))
    }
}