[Accepted] SE-0167: Swift Encoders

Evolution Announcements
1

Proposal Link: swift-evolution/0167-swift-encoders.md at master · apple/swift-evolution · GitHub

The review of SE-0167 "SE-0167: Swift Encoders” ran from April 6...12, 2017. The proposal is accepted. Thanks to everyone who participated in the review!

  - Doug
  Review Manager

2

Hi,

Sorry, I'm very late to the party but have one quick question that I think should be resolved/documented before the patch is landed:

What do we do with integers outside the range [-(2**53)+1, (2**53)-1])? Those are which are the integers that are precisely representable by doubles (IEEE 754-2008 binary64 (double precision). And the problem is that at least JavaScript treats all numbers as doubles, which leads to this problem:

[9851624185071827, 9851624185071829] as [Double] makes them
[9851624185071828, 9851624185071828]

(so two different numbers get both mapped to a third number which sometimes causes problems in the real world [4])

The I(nternet)-JSON RFC [1] states that

   Implementations that generate I-JSON messages cannot assume that
   receiving implementations can process numeric values with greater
   magnitude or precision than provided by those numbers.

Now since Swift isn't JavaScript we fortunately don't store all numbers as doubles so I'm sure a roundtrip of the number 9851624185071827 (which is outside that range) will just work. Nevertheless the RFC [1] says

   For applications that require the exact interchange of numbers with
   greater magnitude or precision, it is RECOMMENDED to encode them in
   JSON string values.

I'm not sure if following that recommendation is a good idea but in any case I think it would be worth documenting it. Other encoders sometimes allow you to specify 'numbers as strings' as an option [2] or outright refuse to encode it.

Twitter also covers the subject [3] for its API.

···

--
  Johannes

[1]: RFC 7493 - The I-JSON Message Format
[2]: JsonGenerator.Feature (Jackson-core 2.4.0 API)
[3]: Twitter IDs | Docs | Twitter Developer Platform
[4]: https://github.com/nodejs/node/issues/12115

On 26 Apr 2017, at 12:10 am, Douglas Gregor via swift-evolution <swift-evolution@swift.org> wrote:

Proposal Link: https://github.com/apple/swift-evolution/blob/master/proposals/0167-swift-encoders.md

The review of SE-0167 "SE-0167: Swift Encoders” ran from April 6...12, 2017. The proposal is accepted. Thanks to everyone who participated in the review!

  - Doug
  Review Manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

3

Hi Johannes,

This is implementation detail that is subject to change, but JSONEncoder and JSONDecoder defer to JSONSerialization to provide the actual serialization to and from JSON.
Internally, numbers are represented in NSNumber instances until you ask for them on decode, so large integers are indeed non-lossy — you can round-trip the values in a 64-bit integer just fine.

If you ask to coerce them as Double values, they will coerce (and lose precision), but that is true today.
Of course, it is not possible to encode values out of Double range as Doubles, since you cannot construct such a Double value.

To wit, any number value encoded via JSONEncoder will always be round-trippable via JSONDecoder provided that you try to decode as the same type as you encoded; this we guarantee.

tl;dr: If you ask for this number as an Int64 or UInt64, you will get the full number without loss of precision.

— Itai

···

On May 9, 2017, at 9:27 AM, Johannes Weiss via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

Sorry, I'm very late to the party but have one quick question that I think should be resolved/documented before the patch is landed:

What do we do with integers outside the range [-(2**53)+1, (2**53)-1])? Those are which are the integers that are precisely representable by doubles (IEEE 754-2008 binary64 (double precision). And the problem is that at least JavaScript treats all numbers as doubles, which leads to this problem:

[9851624185071827, 9851624185071829] as [Double] makes them
[9851624185071828, 9851624185071828]

(so two different numbers get both mapped to a third number which sometimes causes problems in the real world [4])

The I(nternet)-JSON RFC [1] states that

  Implementations that generate I-JSON messages cannot assume that
  receiving implementations can process numeric values with greater
  magnitude or precision than provided by those numbers.

Now since Swift isn't JavaScript we fortunately don't store all numbers as doubles so I'm sure a roundtrip of the number 9851624185071827 (which is outside that range) will just work. Nevertheless the RFC [1] says

  For applications that require the exact interchange of numbers with
  greater magnitude or precision, it is RECOMMENDED to encode them in
  JSON string values.

I'm not sure if following that recommendation is a good idea but in any case I think it would be worth documenting it. Other encoders sometimes allow you to specify 'numbers as strings' as an option [2] or outright refuse to encode it.

Twitter also covers the subject [3] for its API.

--
Johannes

[1]: RFC 7493 - The I-JSON Message Format
[2]: JsonGenerator.Feature (Jackson-core 2.4.0 API)
[3]: Twitter IDs | Docs | Twitter Developer Platform
[4]: Node occasionally gives multiple files/folders the same inode · Issue #12115 · nodejs/node · GitHub

On 26 Apr 2017, at 12:10 am, Douglas Gregor via swift-evolution <swift-evolution@swift.org> wrote:

Proposal Link: https://github.com/apple/swift-evolution/blob/master/proposals/0167-swift-encoders.md

The review of SE-0167 "SE-0167: Swift Encoders” ran from April 6...12, 2017. The proposal is accepted. Thanks to everyone who participated in the review!

  - Doug
  Review Manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

4

Hi Itai,

Hi Johannes,

This is implementation detail that is subject to change, but JSONEncoder and JSONDecoder defer to JSONSerialization to provide the actual serialization to and from JSON.
Internally, numbers are represented in NSNumber instances until you ask for them on decode, so large integers are indeed non-lossy — you can round-trip the values in a 64-bit integer just fine.

thanks! I just thought we should at least document it because I guess JSON with very large integers cannot be used reliably everywhere (as JS for example doesn't handle that). So might be worth making people designing APIs aware of that, no?

-- Johannes

···

On 9 May 2017, at 9:05 pm, Itai Ferber <iferber@apple.com> wrote:

If you ask to coerce them as Double values, they will coerce (and lose precision), but that is true today.
Of course, it is not possible to encode values out of Double range as Doubles, since you cannot construct such a Double value.

To wit, any number value encoded via JSONEncoder will always be round-trippable via JSONDecoder provided that you try to decode as the same type as you encoded; this we guarantee.

tl;dr: If you ask for this number as an Int64 or UInt64, you will get the full number without loss of precision.

— Itai

On May 9, 2017, at 9:27 AM, Johannes Weiss via swift-evolution <swift-evolution@swift.org> wrote:

Hi,

Sorry, I'm very late to the party but have one quick question that I think should be resolved/documented before the patch is landed:

What do we do with integers outside the range [-(2**53)+1, (2**53)-1])? Those are which are the integers that are precisely representable by doubles (IEEE 754-2008 binary64 (double precision). And the problem is that at least JavaScript treats all numbers as doubles, which leads to this problem:

[9851624185071827, 9851624185071829] as [Double] makes them
[9851624185071828, 9851624185071828]

(so two different numbers get both mapped to a third number which sometimes causes problems in the real world [4])

The I(nternet)-JSON RFC [1] states that

  Implementations that generate I-JSON messages cannot assume that
  receiving implementations can process numeric values with greater
  magnitude or precision than provided by those numbers.

Now since Swift isn't JavaScript we fortunately don't store all numbers as doubles so I'm sure a roundtrip of the number 9851624185071827 (which is outside that range) will just work. Nevertheless the RFC [1] says

  For applications that require the exact interchange of numbers with
  greater magnitude or precision, it is RECOMMENDED to encode them in
  JSON string values.

I'm not sure if following that recommendation is a good idea but in any case I think it would be worth documenting it. Other encoders sometimes allow you to specify 'numbers as strings' as an option [2] or outright refuse to encode it.

Twitter also covers the subject [3] for its API.

--
Johannes

[1]: RFC 7493 - The I-JSON Message Format
[2]: JsonGenerator.Feature (Jackson-core 2.4.0 API)
[3]: Twitter IDs | Docs | Twitter Developer Platform
[4]: Node occasionally gives multiple files/folders the same inode · Issue #12115 · nodejs/node · GitHub

On 26 Apr 2017, at 12:10 am, Douglas Gregor via swift-evolution <swift-evolution@swift.org> wrote:

Proposal Link: https://github.com/apple/swift-evolution/blob/master/proposals/0167-swift-encoders.md

The review of SE-0167 "SE-0167: Swift Encoders” ran from April 6...12, 2017. The proposal is accepted. Thanks to everyone who participated in the review!

  - Doug
  Review Manager

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution