PITCH: Export _JSONEncoder / _JSONDecoder

At Swift Summit, we discussed with Joe and Jordan about opening up the Encoder/Decoder classes in order to make the work of an encoder designer easier.

As I was working on an API project, I found myself into the situation of needing to tweak so slightly the encoding strategy this required a full copy/paste of the JSONEncoder.swift file and playing with the internals. I also wanted to implement a simple QueryStringEncoder/Decoder that would properly encode / decode a query string.

The internally defined classes are proven a very powerful tool of reflection as well, being able to collect / untransform a series of containers safely into a strongly typed swift object.

The pitch:

- Keep JSONEncoder / JSONDecoder as 'proxies' to encoding to Data
- Make _JSONEncoder / _JSONDecoder open classes
- Mark public all container implementations of UnkeyedEncodingContainers etc...
- Find a good naming for the _JSONEncoder and _JSONDecoder, that doesn't conflict with JSONEncoder / JSONDecoder but also denotes they conform to Encoder.

Opening those API's isn't for the general Codable implementation, the JSONEncoder/JSONDecoder should stay as-is but it's intended to reduce the amount of boiler plate one would need to implement in order to provide different serialization mechanism / strategy.

1 Like

Hi Florent,

We definitely thought about this while designing the set of types with the Codable proposals.

One serious concern was just how much API surface area there already is with Codable. If we open up the internal classes as well, we risk confusing the majority of people who are just adopting Codable with APIs that are intended only for the minority of people who are trying to create their own custom encoders and decoders.

Any thoughts on how to mitigate this?

- Tony

···

On Nov 3, 2017, at 9:58 AM, Florent Vilmart via swift-evolution <swift-evolution@swift.org> wrote:

At Swift Summit, we discussed with Joe and Jordan about opening up the Encoder/Decoder classes in order to make the work of an encoder designer easier.

As I was working on an API project, I found myself into the situation of needing to tweak so slightly the encoding strategy this required a full copy/paste of the JSONEncoder.swift file and playing with the internals. I also wanted to implement a simple QueryStringEncoder/Decoder that would properly encode / decode a query string.

The internally defined classes are proven a very powerful tool of reflection as well, being able to collect / untransform a series of containers safely into a strongly typed swift object.

The pitch:

- Keep JSONEncoder / JSONDecoder as 'proxies' to encoding to Data
- Make _JSONEncoder / _JSONDecoder open classes
- Mark public all container implementations of UnkeyedEncodingContainers etc...
- Find a good naming for the _JSONEncoder and _JSONDecoder, that doesn't conflict with JSONEncoder / JSONDecoder but also denotes they conform to Encoder.

Opening those API's isn't for the general Codable implementation, the JSONEncoder/JSONDecoder should stay as-is but it's intended to reduce the amount of boiler plate one would need to implement in order to provide different serialization mechanism / strategy.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Hi Florent,

We definitely thought about this while designing the set of types with the Codable proposals.

One serious concern was just how much API surface area there already is with Codable. If we open up the internal classes as well, we risk confusing the majority of people who are just adopting Codable with APIs that are intended only for the minority of people who are trying to create their own custom encoders and decoders.

Any thoughts on how to mitigate this?

- Tony

I’ve been curious for some time about if we can do something about an opt-in import in Swift?

For example, currently UIGestureRecognizer in UIKit has “subclass only” methods that are protected and opt in. Importing UIKit itself doesn’t bring it in, and instead you need to specifically import UIKit.UIGestureRecognizerSubclass.

I realise this is a standalone case, but I’m wondering whether we can generalise this into something we can propose, to actively support nested scopes in the same way?

This would lend well to disclosing the internals like this. It would avoid users jumping straight for the internal types because they wouldn’t be there with “import Foundation” - it would require something eg “import Foundation.xyz <http://foundation.xyz/>”.

Was this part of an earlier discussion of modules etc?

- Rod

···

On 7 Nov 2017, at 6:24 am, Tony Parker via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 3, 2017, at 9:58 AM, Florent Vilmart via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

At Swift Summit, we discussed with Joe and Jordan about opening up the Encoder/Decoder classes in order to make the work of an encoder designer easier.

As I was working on an API project, I found myself into the situation of needing to tweak so slightly the encoding strategy this required a full copy/paste of the JSONEncoder.swift file and playing with the internals. I also wanted to implement a simple QueryStringEncoder/Decoder that would properly encode / decode a query string.

The internally defined classes are proven a very powerful tool of reflection as well, being able to collect / untransform a series of containers safely into a strongly typed swift object.

The pitch:

- Keep JSONEncoder / JSONDecoder as 'proxies' to encoding to Data
- Make _JSONEncoder / _JSONDecoder open classes
- Mark public all container implementations of UnkeyedEncodingContainers etc...
- Find a good naming for the _JSONEncoder and _JSONDecoder, that doesn't conflict with JSONEncoder / JSONDecoder but also denotes they conform to Encoder.

Opening those API's isn't for the general Codable implementation, the JSONEncoder/JSONDecoder should stay as-is but it's intended to reduce the amount of boiler plate one would need to implement in order to provide different serialization mechanism / strategy.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I really like this idea of scoped import, but I’m not sure it’s part of anything supported widely. I was about to suggest ‘namespaces’ but that’s not a thing in swift.

···

On Nov 9, 2017, 17:06 -0500, Rod Brown <rodney.brown6@icloud.com>, wrote:

> On 7 Nov 2017, at 6:24 am, Tony Parker via swift-evolution <swift-evolution@swift.org> wrote:
>
> Hi Florent,
>
> We definitely thought about this while designing the set of types with the Codable proposals.
>
> One serious concern was just how much API surface area there already is with Codable. If we open up the internal classes as well, we risk confusing the majority of people who are just adopting Codable with APIs that are intended only for the minority of people who are trying to create their own custom encoders and decoders.
>
> Any thoughts on how to mitigate this?
>
> - Tony

I’ve been curious for some time about if we can do something about an opt-in import in Swift?

For example, currently UIGestureRecognizer in UIKit has “subclass only” methods that are protected and opt in. Importing UIKit itself doesn’t bring it in, and instead you need to specifically import UIKit.UIGestureRecognizerSubclass.

I realise this is a standalone case, but I’m wondering whether we can generalise this into something we can propose, to actively support nested scopes in the same way?

This would lend well to disclosing the internals like this. It would avoid users jumping straight for the internal types because they wouldn’t be there with “import Foundation” - it would require something eg “import Foundation.xyz”.

Was this part of an earlier discussion of modules etc?

- Rod

>
> > On Nov 3, 2017, at 9:58 AM, Florent Vilmart via swift-evolution <swift-evolution@swift.org> wrote:
> >
> > At Swift Summit, we discussed with Joe and Jordan about opening up the Encoder/Decoder classes in order to make the work of an encoder designer easier.
> >
> > As I was working on an API project, I found myself into the situation of needing to tweak so slightly the encoding strategy this required a full copy/paste of the JSONEncoder.swift file and playing with the internals. I also wanted to implement a simple QueryStringEncoder/Decoder that would properly encode / decode a query string.
> >
> > The internally defined classes are proven a very powerful tool of reflection as well, being able to collect / untransform a series of containers safely into a strongly typed swift object.
> >
> > The pitch:
> >
> > - Keep JSONEncoder / JSONDecoder as 'proxies' to encoding to Data
> > - Make _JSONEncoder / _JSONDecoder open classes
> > - Mark public all container implementations of UnkeyedEncodingContainers etc...
> > - Find a good naming for the _JSONEncoder and _JSONDecoder, that doesn't conflict with JSONEncoder / JSONDecoder but also denotes they conform to Encoder.
> >
> > Opening those API's isn't for the general Codable implementation, the JSONEncoder/JSONDecoder should stay as-is but it's intended to reduce the amount of boiler plate one would need to implement in order to provide different serialization mechanism / strategy.
> >
> >
> > _______________________________________________
> > swift-evolution mailing list
> > swift-evolution@swift.org
> > https://lists.swift.org/mailman/listinfo/swift-evolution
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

Beyond the question of how to integrate this into Foundation, I would like to suggest a concrete idea about how the open implementation could/should look like.

The strategy I’m going to describe is already implemented and tested up to some point as MetaSerialization here.

As I tried to implement a Encoder and Decoder for msgpack, I realised, that the main work of _JSONEncoder seems to be to convert it’s Codable input to a representation that JSONSerialization is able to work with. The critical point here is in my understanding, that other underlying serialization libraries will need different representations and may often want to avoid NSDictionary, NSArray, NSString, etc, for which the implementor would still need to override large parts of the open Encoder implementation. But how this representation looks like is mainly related to the “primitive” types of the serialization format. Those primitive types are also the types the Encoder needs to convert to and the Decoder needs to convert from. The code that is involved in this are the various wrap and unwrap functions.

(Note that there is also some code especially in the unwrap function, that should be common to most implementations: Detecting instances that aren’t able to decode itself and throw an error for these instances, to prevent endless loops)

Now what I did in MetaSerialization is providing MetaEncoder and MetaDecoder (as open classes, for further customisation) that handle the coding storage (called CodingStack in MetaSerialization) and the various container methods (there are therefor also open implementations for these container protocols). MetaEncoder and MetaDecoder rely on an instance of the protocol Translator, that the user of the framework needs to provide. This Translator defines the primitive types and handles the encoding/decoding step. The current implementation requires it to implement four methods (two of which have defaults) in order to set and handle primitive types. wrappingMeta returns a wrapper for a certain type in the encoding step, if this type is primitive (the wrapper is called Meta and needs to conform to protocol Meta) or returns nil, if a type isn’t primitive. MetaEncoder calls this function for every value it sees and directly pushes this meta to it’s coding stack, or continues to encode the value, if it get’s nil. Finally all types need to be converted to primitive types, or (un)keyed containers of primitive types. The implementations that are actually used for these containers are also set by the Translator in two functions called keyedContainer and unkeyedContainer.
When decoding, MetaDecoder asks the Translator to unwrap a certain Meta to a certain type. Translator may return nil, if it can’t unwrap to this type, or returns the wrapped value, if the type can be converted from the stored. MetaDecoder detects types that produce endless loops, because they aren’t supported but have not strategy to decode themselves from any other representation.

These ideas and the implementation of them isn’t finished yet and I’m not entirely fine with the Translator protocol, because I think it could be simpler.

A simple example, that demonstrates a verry simple way to use meta-serialization can be found in the playground contained in the repository. However, this example does not show how to implement Translator. There is another example in a separate repository that implements Translator and also shows concrete imolementations and use cases for custom implementations of Meta.

I hope that this description gives a new idea about it could be implemented.

2 Likes

This looks interesting. I will definitely take a look.