[Pitch] JSON Encoding Int64 as String

JSONEncoder encodes Int64 and UInt64 values as numbers (no quotes), but this is incompatible with JavaScript as JavaScript only supports integer values up to 53-bits long. This means that iOS-generated JSON is incompatible (or lossy) with back ends written in node.js, or generally any JavaScript recipient.

There are several JSON specifications, such as Protocol Buffers, Cap'n Proto, and Google's API Discovery Service which standardize on converting all 64-bit ints into strings in JSON. This option, however, is not easily achievable using Swift's JSONEncoder.

I propose to add an option similar to NonConformingFloatEncodingStrategy, such as an Int64EncodingStrategy, which provides an option to encode all 64-bit integers as strings.

Hi Eric,
You're quite right about the formatting woes with large integers.

I would suggest though trying to solve it using an alternative approach rather than putting more smarts into Foundation's JSON coder, specifically, you can solve it today via such property wrapper:


    @propertyWrapper
    public struct StringRepresentedInt<IntegerType: FixedWidthInteger>: Codable {
        public var wrappedValue: IntegerType

        public init(wrappedValue: IntegerType) {
            self.wrappedValue = wrappedValue
        }

        public init(from decoder: Decoder) throws {
            let container = try decoder.singleValueContainer()
            let stringValue = try container.decode(String.self)
            guard let value = IntegerType(stringValue) else {
                fatalError("Could not parse string representation of integer [\(stringValue)] as \(IntegerType.self)!")
            }
            self.wrappedValue = value
        }

        public func encode(to encoder: Encoder) throws {
            var container = encoder.singleValueContainer()
            try container.encode("\(self.wrappedValue)")
        }
    }
    struct Hello: Codable {
        @StringRepresentedInt
        var number: Int64
    }
    func test_coding_StringRepresentedInt() throws {
        let hello = Hello(number: 128)
        let data = try JSONEncoder().encode(hello)
        let back = try JSONDecoder().decode(Hello.self, from: data)

        print("data = \(data.stringDebugDescription())") // data = {"number":"128"}
        XCTAssertEqual(hello.number, back.number)
    }

following the general idea / shape of what is being done in this repo: GitHub - marksands/BetterCodable: Better Codable through Property Wrappers

Feel free to take the snippet polish it up a bit and perhaps it's worth contributing to Better Codable if you'd want to so others hitting the same issue may benefit as well?

3 Likes

Heh, this is exactly how we have it solved right now, and it's viable, but it feels pretty clunky. Some other considerations:

  • You need additional variants for arrays and dictionaries as well, which starts to be a bit more code to copy/paste into multiple projects. This applies to arrays of arrays (of arrays) of 64-bit numbers, etc as well.
  • The coupling is wrong...the encoding format should be a property of the encoder and not the type being encoded. As a clarifying example, if I wanted to send the same data to a JS backend using string numbers, and a Java backend using integer numbers, I now need two model definitions.
1 Like

You could make a wrapper type StringRepresentedInt(myInt) that'd be stored in those properties. (And yes, I very much realize then the "format" leaked into the "model" as well).

But yeah, Codable today is not very "controllable" or "customizable", not going to argue against that.
Optimally we'd have some general way to allow customizing per field such things.

Realistically what seems to be a widely used way to solve this kerfuffle seen APIs do "out in the wild" is to offer both, e.g. id and id_str: Tweet object | Docs | Twitter Developer Platform

But yeah, if corelibs folks would want to adopt this additional setting that's also an option.

Having it as a corelib option definitely feels like the cleanest and most consistent path for something that feels like a concrete edge case that isn't handled today. There's admittedly a workaround, but per the conversation above it did start to spiral into complicated territory pretty quickly.

I like the id/id_str method for public APIs. For our use case we're trying to implement the Protocol Buffers JSON spec, which requires unilaterally encoding Int64s as strings, so that method doesn't quite fit the bill.

Thanks for the thoughts @ktoso (and the code snippets; they're cleaner than mine). Curious to hear what the corelib team thinks.

2 Likes

I think this would be a decently pragmatic addition to JSONEncoder and JSONDecoder, especially if you're encoding types you don't own and can't manually encode their values as strings. However, since Foundation API is not part of Swift evolution, your best bet would be to file feedback with this idea and some examples.

Thanks for the idea @efirestone. I'll track it for Foundation here: rdar://problem/65148569

Out of curiosity, how does the decode side work? An option to allow a string to be decoded as a number?

Thanks for writing it up! For the decode, I'm hand waving a bit because I'm not familiar with the current JSONDecoder internals, but I presume it would be automatic based on the type of the property being set (determined, I assume, through reflection). So I wouldn't expect an option is needed for the decode side, and that the decoder would do something like:

if decodedTypeField is Int64 {
   if JSONValue is number
      convert JSON number to Int64
   } else if JSONValue is string {
      attempt to convert string to Int64, else throw type error
   }
}

I just ran across this exact same scenario today as well. We recently stripped out the Protobuf runtime out of our SDK and opted to just deal with the JSON between the client and the backend. All of the Structs we expose to users of our SDK are auto generated based off the Protobuf definitions.

Today I started seeing this issue with the strings being returned instead of Int64s now that one request is returning the entire data for the request instead of the subset that it used to.

Looking into it, yes, the Protobuf spec states that Int64 will be serialized to Strings in JSON.

The unfortunate part I am facing now is that the definitions of the codable structs are using immutable values (let) so I can't use property wrappers.

Looking for another solution that doesn't involve me having to write customer encoders and decoders for each struct in the data model.