Decode a JSON object of unknown format into a Dictionary with Decodable in Swift 4


(Kevin Wooten) #1

Hi Jon,

I just joined this mailing list and have tried to catch up on the
history of this thread, so please excuse me if I’ve missed something.

I’m sorry the Codable API at the moment does not answer your needs —
you’re clearly not the only one who’s run into this, so let’s see
how we can work together to make the API better for everyone.
For one thing, in the case of grabbing a subtree of JSON as
"unevaluated" or "unmapped" (as it appears to be in the metadata case),
it should be fairly simple to add a `JSONDecoder.UnevaluatedJSON` type
that will allow you to essentially decode that part of the tree as an
`Any`. `JSONDecoder` would have knowledge of this type and would be able
to return the subtree inside of it — you’d decode a property as
`JSONDecoder.UnevaluatedJSON.self` and access the contents through `var
value: Any?`, or something similar. This would be simple additive API,
which although might not make it in the upcoming betas, should be fairly
simple introduce. Would this solve that use case?

We’re also working on improving `NSISO8601DateFormatter`. I don’t
think I saw it in any of your emails — what specific use case are you
looking for that it doesn’t at the moment support?

— Itai

Itai,

Is this a formal solution that is going to be implemented? This would solve just about every issue I currently have with Decodable.

Two points…

1) Putting it on `JSONDecoder` seems dubious since you’d only have access to `Decoder` (although conditional casting could solve that). It seems adding the method to `Decoder` and using `Decoder.Unevaluated.self` as the requested type, would be more useful. A user could then conditionally cast that value to things like `[String: Any]` and possibly use its contents generically.

2) Matching it with an equivalent on `Encoder` would be great as well. We take in JSON that has “metaData” like one aforementioned exampled. We then have to send back the equivalent metadata during a subsequent update; without ever inspecting or altering the unevaluated data. Being able encode a `Decoder.Unevaluated` would solve that problem as well.


Possible 4.1 Encodable regression? (+ segmentation fault)
(Itai Ferber) #2

Hi Kevin,

Hi Jon,

I just joined this mailing list and have tried to catch up on the
history of this thread, so please excuse me if I’ve missed something.

I’m sorry the Codable API at the moment does not answer your needs —
you’re clearly not the only one who’s run into this, so let’s see
how we can work together to make the API better for everyone.
For one thing, in the case of grabbing a subtree of JSON as
"unevaluated" or "unmapped" (as it appears to be in the metadata case),
it should be fairly simple to add a `JSONDecoder.UnevaluatedJSON` type
that will allow you to essentially decode that part of the tree as an
`Any`. `JSONDecoder` would have knowledge of this type and would be able
to return the subtree inside of it — you’d decode a property as
`JSONDecoder.UnevaluatedJSON.self` and access the contents through `var
value: Any?`, or something similar. This would be simple additive API,
which although might not make it in the upcoming betas, should be fairly
simple introduce. Would this solve that use case?

We’re also working on improving `NSISO8601DateFormatter`. I don’t
think I saw it in any of your emails — what specific use case are you
looking for that it doesn’t at the moment support?

— Itai

Itai,

Is this a formal solution that is going to be implemented? This would solve just about every issue I currently have with Decodable.

I can’t make any promises at the moment — we’ve got a lot of high-priority things to fix before the Swift 4.0 release. However, this is something I’d certainly like to put through API review and eventually release, since this is clearly something that would be beneficial to a lot of our users.

Two points…

1) Putting it on `JSONDecoder` seems dubious since you’d only have access to `Decoder` (although conditional casting could solve that). It seems adding the method to `Decoder` and using `Decoder.Unevaluated.self` as the requested type, would be more useful. A user could then conditionally cast that value to things like `[String: Any]` and possibly use its contents generically.

Putting that on Decoder would require all Decoders to have an "unevaluated type" representation, which may not be appropriate for all formats.
Since this is very often a request when working with 3rd-party APIs which you don’t control (and are rarely offered in more than one format, if that), putting this directly on JSONDecoder seems reasonable — you’d only really expect this representation if you’re decoding from JSON; if you’re encoding to/from a different format, you’re likely in control of the data in those formats.

2) Matching it with an equivalent on `Encoder` would be great as well. We take in JSON that has “metaData” like one aforementioned exampled. We then have to send back the equivalent metadata during a subsequent update; without ever inspecting or altering the unevaluated data. Being able encode a `Decoder.Unevaluated` would solve that problem as well.

Yes, we’d add an equivalent type on encode as well.

···

On Jun 29, 2017, at 12:30 PM, Kevin Wooten via swift-users <swift-users@swift.org> wrote:

_______________________________________________
swift-users mailing list
swift-users@swift.org
https://lists.swift.org/mailman/listinfo/swift-users


(Kevin Wooten) #3

Itai,

I tried copying JSONEncoder.swift from the swift repo and implementing the `Unevaluated` type in it. It doesn’t appear to be a workable solution due to the fact that the `KeyedEncodingContainer` type erasure box is the only value returned from `container(keyedBy:)`. This means that any extra methods implemented inside `JSONDecoder` (or `_JSONDecoder` in this case) can ever be publicly accessed (even with casting) unless the type erasure box also includes an equivalent method.

How do you propose getting around this? Seems like the design of this might be a bit limiting when considering extension/enhancement in user specific code.

···

On Jun 29, 2017, at 12:47 PM, Itai Ferber <iferber@apple.com> wrote:

Hi Kevin,

On Jun 29, 2017, at 12:30 PM, Kevin Wooten via swift-users <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Hi Jon,

I just joined this mailing list and have tried to catch up on the
history of this thread, so please excuse me if I’ve missed something.

I’m sorry the Codable API at the moment does not answer your needs —
you’re clearly not the only one who’s run into this, so let’s see
how we can work together to make the API better for everyone.
For one thing, in the case of grabbing a subtree of JSON as
"unevaluated" or "unmapped" (as it appears to be in the metadata case),
it should be fairly simple to add a `JSONDecoder.UnevaluatedJSON` type
that will allow you to essentially decode that part of the tree as an
`Any`. `JSONDecoder` would have knowledge of this type and would be able
to return the subtree inside of it — you’d decode a property as
`JSONDecoder.UnevaluatedJSON.self` and access the contents through `var
value: Any?`, or something similar. This would be simple additive API,
which although might not make it in the upcoming betas, should be fairly
simple introduce. Would this solve that use case?

We’re also working on improving `NSISO8601DateFormatter`. I don’t
think I saw it in any of your emails — what specific use case are you
looking for that it doesn’t at the moment support?

— Itai

Itai,

Is this a formal solution that is going to be implemented? This would solve just about every issue I currently have with Decodable.

I can’t make any promises at the moment — we’ve got a lot of high-priority things to fix before the Swift 4.0 release. However, this is something I’d certainly like to put through API review and eventually release, since this is clearly something that would be beneficial to a lot of our users.

Two points…

1) Putting it on `JSONDecoder` seems dubious since you’d only have access to `Decoder` (although conditional casting could solve that). It seems adding the method to `Decoder` and using `Decoder.Unevaluated.self` as the requested type, would be more useful. A user could then conditionally cast that value to things like `[String: Any]` and possibly use its contents generically.

Putting that on Decoder would require all Decoders to have an "unevaluated type" representation, which may not be appropriate for all formats.
Since this is very often a request when working with 3rd-party APIs which you don’t control (and are rarely offered in more than one format, if that), putting this directly on JSONDecoder seems reasonable — you’d only really expect this representation if you’re decoding from JSON; if you’re encoding to/from a different format, you’re likely in control of the data in those formats.

2) Matching it with an equivalent on `Encoder` would be great as well. We take in JSON that has “metaData” like one aforementioned exampled. We then have to send back the equivalent metadata during a subsequent update; without ever inspecting or altering the unevaluated data. Being able encode a `Decoder.Unevaluated` would solve that problem as well.

Yes, we’d add an equivalent type on encode as well.

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users


(Itai Ferber) #4

Hi Kevin,

You’re right — this is one of the limitations of the box design here. One thing we can do is expose the underlying boxed value as a `KeyedDecodingContainerProtocol` existential value using an accessor on the box so you can down-cast.

However, it shouldn’t be necessary to add any methods to `_JSONEncoder` or `_JSONDecoder` to support this. What are you trying to do?

— Itai

···

On 1 Jul 2017, at 9:07, Kevin Wooten wrote:

Itai,

I tried copying JSONEncoder.swift from the swift repo and implementing the `Unevaluated` type in it. It doesn’t appear to be a workable solution due to the fact that the `KeyedEncodingContainer` type erasure box is the only value returned from `container(keyedBy:)`. This means that any extra methods implemented inside `JSONDecoder` (or `_JSONDecoder` in this case) can ever be publicly accessed (even with casting) unless the type erasure box also includes an equivalent method.

How do you propose getting around this? Seems like the design of this might be a bit limiting when considering extension/enhancement in user specific code.

On Jun 29, 2017, at 12:47 PM, Itai Ferber <iferber@apple.com> wrote:

Hi Kevin,

On Jun 29, 2017, at 12:30 PM, Kevin Wooten via swift-users >>> <swift-users@swift.org <mailto:swift-users@swift.org>> wrote:

Hi Jon,

I just joined this mailing list and have tried to catch up on the
history of this thread, so please excuse me if I’ve missed something.

I’m sorry the Codable API at the moment does not answer your needs —
you’re clearly not the only one who’s run into this, so let’s see
how we can work together to make the API better for everyone.
For one thing, in the case of grabbing a subtree of JSON as
"unevaluated" or "unmapped" (as it appears to be in the metadata case),
it should be fairly simple to add a `JSONDecoder.UnevaluatedJSON` type
that will allow you to essentially decode that part of the tree as an
`Any`. `JSONDecoder` would have knowledge of this type and would be able
to return the subtree inside of it — you’d decode a property as
`JSONDecoder.UnevaluatedJSON.self` and access the contents through `var
value: Any?`, or something similar. This would be simple additive API,
which although might not make it in the upcoming betas, should be fairly
simple introduce. Would this solve that use case?

We’re also working on improving `NSISO8601DateFormatter`. I don’t
think I saw it in any of your emails — what specific use case are you
looking for that it doesn’t at the moment support?

— Itai

Itai,

Is this a formal solution that is going to be implemented? This would solve just about every issue I currently have with Decodable.

I can’t make any promises at the moment — we’ve got a lot of high-priority things to fix before the Swift 4.0 release. However, this is something I’d certainly like to put through API review and eventually release, since this is clearly something that would be beneficial to a lot of our users.

Two points…

1) Putting it on `JSONDecoder` seems dubious since you’d only have access to `Decoder` (although conditional casting could solve that). It seems adding the method to `Decoder` and using `Decoder.Unevaluated.self` as the requested type, would be more useful. A user could then conditionally cast that value to things like `[String: Any]` and possibly use its contents generically.

Putting that on Decoder would require all Decoders to have an "unevaluated type" representation, which may not be appropriate for all formats.
Since this is very often a request when working with 3rd-party APIs which you don’t control (and are rarely offered in more than one format, if that), putting this directly on JSONDecoder seems reasonable — you’d only really expect this representation if you’re decoding from JSON; if you’re encoding to/from a different format, you’re likely in control of the data in those formats.

2) Matching it with an equivalent on `Encoder` would be great as well. We take in JSON that has “metaData” like one aforementioned exampled. We then have to send back the equivalent metadata during a subsequent update; without ever inspecting or altering the unevaluated data. Being able encode a `Decoder.Unevaluated` would solve that problem as well.

Yes, we’d add an equivalent type on encode as well.

_______________________________________________
swift-users mailing list
swift-users@swift.org <mailto:swift-users@swift.org>
https://lists.swift.org/mailman/listinfo/swift-users